The Cloud(txt.black)
txt.black
The Cloud
https://txt.black/~jack/cloud.txt
165 comments
I wouldn't go so far as the author when recommending tech stacks for startups or hobby projects. The main benefit of cloud providers is that you don't have to learn and pay attention to all the stuff he lists, and you can focus on building the software you want to build.
But I do think he's performing a useful service in highlighting just how much you're paying for not having to hire a devops or sysadmin person. 10x performance differentials have been replicated not just by him but in other benchmarks. I remember seeing a chart when I was still working at Google and GCP was being justified that showed a graph of hard disk prices vs. S3 rates - since inception (2007-2012 at this time), S3 rates had gone down by a factor of about 2-3x, but price/GB of physical hard disk space had gone down by ~100x. Amazon is hiding all of this improvement in the physical hardware behind vCPUs and opaque billing, and it becomes pure profit to them.
I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins. At that level you could easily afford them, and you probably can get old Linux greybeards cheap now that everyone thinks the job description is obsolete.
But I do think he's performing a useful service in highlighting just how much you're paying for not having to hire a devops or sysadmin person. 10x performance differentials have been replicated not just by him but in other benchmarks. I remember seeing a chart when I was still working at Google and GCP was being justified that showed a graph of hard disk prices vs. S3 rates - since inception (2007-2012 at this time), S3 rates had gone down by a factor of about 2-3x, but price/GB of physical hard disk space had gone down by ~100x. Amazon is hiding all of this improvement in the physical hardware behind vCPUs and opaque billing, and it becomes pure profit to them.
I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins. At that level you could easily afford them, and you probably can get old Linux greybeards cheap now that everyone thinks the job description is obsolete.
> I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins.
I'm not surprised at all: that's thanks to the beauty of lock-in effects. Especially all those cloud-specific managed services make it really hard to switch, and even if you are treating the cloud as a pure VM platform, it is considerable work and risk to switch to a self-managed bare metal setup.
Basically, if you don't start bare-metal (incurring having to assemble the necessary know-how to do so) you will have a hard time to get clean from the cloud drug later on, when you think you're ready to afford having the know-how. I think this aspect has been a main driver behind the linked article.
I'm not surprised at all: that's thanks to the beauty of lock-in effects. Especially all those cloud-specific managed services make it really hard to switch, and even if you are treating the cloud as a pure VM platform, it is considerable work and risk to switch to a self-managed bare metal setup.
Basically, if you don't start bare-metal (incurring having to assemble the necessary know-how to do so) you will have a hard time to get clean from the cloud drug later on, when you think you're ready to afford having the know-how. I think this aspect has been a main driver behind the linked article.
This is why I'd really like to see projects like Eucalyptus take off, along with (possibly) some standardization of "infrastructure API": https://www.eucalyptus.cloud/index.html
S3 isn’t a hard disk; that’s EBS. S3 is ~17 copies on object-level-mirrored hard disks (allowing a choice of different mirrors per object, preventing overloading and rebuild degradation) attached to a lifecycle event queue, a load-balanced webserver cluster, and an async tape archive.
S3 costs pretty much what that infrastructure costs to run yourself.
But certainly, go ahead and run a Riak CS cluster (plus those other things) on top of some Herzberg servers, and try to outcompete S3 on price. I—and many others—will be waiting. :)
S3 costs pretty much what that infrastructure costs to run yourself.
But certainly, go ahead and run a Riak CS cluster (plus those other things) on top of some Herzberg servers, and try to outcompete S3 on price. I—and many others—will be waiting. :)
BTW, I use a wonderful tool called goofys that lets you mount S3 buckets as local volumes. There are a number of constraints (can open for read, or write a file once) but it works well for a wide range of legacy applications.
"BTW, I use a wonderful tool called goofys that lets you mount S3 buckets as local volumes."
You may be interested in the 'rclone' tool which does all of these things, and more, with all of the cloud providers:
https://rclone.org/
You may be interested in the 'rclone' tool which does all of these things, and more, with all of the cloud providers:
https://rclone.org/
You might want to highlight that rclone has FUSE support – you have to scroll down the page a bit to learn that your comment wasn't comparing unlike things:
https://rclone.org/commands/rclone_mount/
https://rclone.org/commands/rclone_mount/
Does it work better than goofys?
The cost of S3 is pretty dang cheap for advertised 99.99% available, 99.999999999% durable storage: $0.02/GB/month.
And that's ignoring the Glacier storage class (which probably didn't show up on that historical graph), which is a fifth of that.
And that's ignoring the Glacier storage class (which probably didn't show up on that historical graph), which is a fifth of that.
> And that's ignoring the Glacier storage class, which is a fifth of that.
And Glacier Deep Archive is a further fourth of that.
And Glacier Deep Archive is a further fourth of that.
Bandwidth prices vary widely and some are bad, so the “all in” price can be hard to figure. It’s definitely cheap to let your data just sit there. Move it anywhere — it might get expensive.
AWS charges $5 per million write requests, $0.40 per million read requests. Inbound bandwidth is free; outbound bandwidth ranges from free (within same AWS region) to $0.05/GB for public internet at scale.
The latter number can be pricey. But, irrelevant with respect to the failing costs of hard drives.
The latter number can be pricey. But, irrelevant with respect to the failing costs of hard drives.
>The main benefit of cloud providers is that you don't have to learn and pay attention to all the stuff he lists, [...] I'm surprised more mid-size companies - those with AWS bills in the 6-figure-per-month range - don't leave the cloud, get physical hardware, and pay some sysadmins.
$12 million/year for cloud seems like a high enough threshold to go internal but I'm not even sure that's enough money to sway executives away from cloud.
The main issue is that internal IT departments at corporations have slow service from the IT staff compared to AWS/GCP/Azure and its engineers. Corporate IT departments typically don't treat their coworkers in other departments as internal customers. Instead, they treat them as adversaries and a nuisance.
That's why at non-tech corporations, the first group that experimented with the new AWS cloud service were the development teams. They got fed up with IT service backlogs waiting for new dev/sandbox/text servers. They got tired of waiting for IT to requisition a server and then install the os, db, etc. If the programming manager then asked IT to install an Oracle update, the IT department might say "well, Susan our db admin is on vacation till next Monday so we'll get to it then." Those kinds of inter-department interactions were frustrating to the internal customers of the IT department. With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months. Once the dev teams were sold, the more risk-averse and mission-critical production workloads eventually moved to AWS too.
The author of this article doesn't seem to understand the dysfunctional relationship between internal IT and their coworkers they serve. It's more about responsiveness and iteration speed of the AWS/GCP/Azure employees' vs the IT employees than expensive AWS EC2 compute power vs cheaper internal racks of cpu.
E.g. The Guardian pays AWS more for cpu+diskspace+network than what they can build on their own. That's the raw hardware costs. But it doesn't matter because Guardian's internal IT staff with an internal cloud stack just can't match AWS: https://www.computerworld.com/article/3427004/the-guardian-g...
To me, it totally makes sense for Dropbox to migrate off of AWS to save money. They're a tech company and have the engineering culture to do it. A lot of non-tech companies (like The Guardian) don't.
$12 million/year for cloud seems like a high enough threshold to go internal but I'm not even sure that's enough money to sway executives away from cloud.
The main issue is that internal IT departments at corporations have slow service from the IT staff compared to AWS/GCP/Azure and its engineers. Corporate IT departments typically don't treat their coworkers in other departments as internal customers. Instead, they treat them as adversaries and a nuisance.
That's why at non-tech corporations, the first group that experimented with the new AWS cloud service were the development teams. They got fed up with IT service backlogs waiting for new dev/sandbox/text servers. They got tired of waiting for IT to requisition a server and then install the os, db, etc. If the programming manager then asked IT to install an Oracle update, the IT department might say "well, Susan our db admin is on vacation till next Monday so we'll get to it then." Those kinds of inter-department interactions were frustrating to the internal customers of the IT department. With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months. Once the dev teams were sold, the more risk-averse and mission-critical production workloads eventually moved to AWS too.
The author of this article doesn't seem to understand the dysfunctional relationship between internal IT and their coworkers they serve. It's more about responsiveness and iteration speed of the AWS/GCP/Azure employees' vs the IT employees than expensive AWS EC2 compute power vs cheaper internal racks of cpu.
E.g. The Guardian pays AWS more for cpu+diskspace+network than what they can build on their own. That's the raw hardware costs. But it doesn't matter because Guardian's internal IT staff with an internal cloud stack just can't match AWS: https://www.computerworld.com/article/3427004/the-guardian-g...
To me, it totally makes sense for Dropbox to migrate off of AWS to save money. They're a tech company and have the engineering culture to do it. A lot of non-tech companies (like The Guardian) don't.
>Corporate IT groups typically don't treat their non-IT coworkers as customers. Instead, they treat them as adversaries and a nuisance.
One deep problem with the "internal customer" model is that you don't get a larger budget when you serve your "internal customers" better.
One deep problem with the "internal customer" model is that you don't get a larger budget when you serve your "internal customers" better.
> With AWS, the programmers got their sandbox servers in spun up minutes instead of weeks or months.
That's how it starts but not how it ends.
Once you move everything to the cloud, the same IT staff get put in charge of it. Because they have a requirement from somewhere up high to make sure everything has antivirus on it, and there is a Windows domain for the desktops so the servers need to be in it too, and they want to have a test environment with specific versions of specific software matching the ones in production which means you have to use the corporate VM images instead of the AWS ones and have IT go through their VM provisioning checklist for each new instance, etc.
And some of those reasons are bad and some of them aren't, but either way they exist, so if you want to provision something from AWS now it has to go through IT who takes just as long as they ever did to do what amounts to the same thing as before only now it's more expensive.
That's how it starts but not how it ends.
Once you move everything to the cloud, the same IT staff get put in charge of it. Because they have a requirement from somewhere up high to make sure everything has antivirus on it, and there is a Windows domain for the desktops so the servers need to be in it too, and they want to have a test environment with specific versions of specific software matching the ones in production which means you have to use the corporate VM images instead of the AWS ones and have IT go through their VM provisioning checklist for each new instance, etc.
And some of those reasons are bad and some of them aren't, but either way they exist, so if you want to provision something from AWS now it has to go through IT who takes just as long as they ever did to do what amounts to the same thing as before only now it's more expensive.
Counterpoint from someone who did developer support back in the pre-cloud days.
Even pre-virtualization, I could instantiate servers very rapidly with network based imaging. With pre-staged templates of various os/program combinations. Applied them to both lab and production deployment. So that Dev, QA, and Prod could all work from the same base images. With versioning of the images based on patch level.
Then I would subsequently get blamed for every program error by the dev team. With no other justification than "it worked in the lab. Fix your server!" Running a packet analyzer, I would often find things like queries trying to go to a lab db server rather than prod. Or a dependency on an out of date/unpatched/vulnerable version of some supporting software that the development team would then attempt to pressure into production rather than update their code.
For every obstinate sysadmin you've encountered, there is an ops pro who has received a ticket with nothing but a list of compiler errors combined with 'pls fix asap'.
Even pre-virtualization, I could instantiate servers very rapidly with network based imaging. With pre-staged templates of various os/program combinations. Applied them to both lab and production deployment. So that Dev, QA, and Prod could all work from the same base images. With versioning of the images based on patch level.
Then I would subsequently get blamed for every program error by the dev team. With no other justification than "it worked in the lab. Fix your server!" Running a packet analyzer, I would often find things like queries trying to go to a lab db server rather than prod. Or a dependency on an out of date/unpatched/vulnerable version of some supporting software that the development team would then attempt to pressure into production rather than update their code.
For every obstinate sysadmin you've encountered, there is an ops pro who has received a ticket with nothing but a list of compiler errors combined with 'pls fix asap'.
I have replicated the performance differentials (for Azure), and they definitely suck.
But people are REALLY REALLY expensive (not only the ongoing costs, but hiring, firing and overhead), and Cloud has another huge advantage: accounting tricks (which help a lot to said midsize firms).
Literally one of the first things they teach you at the Azure training is the difference between CapEx and OpEx: https://docs.microsoft.com/en-us/learn/paths/azure-fundament...
But people are REALLY REALLY expensive (not only the ongoing costs, but hiring, firing and overhead), and Cloud has another huge advantage: accounting tricks (which help a lot to said midsize firms).
Literally one of the first things they teach you at the Azure training is the difference between CapEx and OpEx: https://docs.microsoft.com/en-us/learn/paths/azure-fundament...
because saying your business is built for and in the cloud gets you investors and street cred. further, if you are having a bad quarter, you can squeeze your infra to lower opex. If you built all onprem, you had to invest capex heavily upfront, and you can't slow down your opex very much in the bad times.
I know the CW is that sysadmin/devops is being automated away, and a lot of people think programming is about automating all the things (or at least as much as you can), but doesn't what you're saying prove that they aren't obsolete for large-scale providers?
I think you’re right. But I think you and the OP are probably coming from different perspectives. Yes if you have frequent scaling issues you probably need the cloud. But a lot of companies shouldn’t have ever have scaling issues and are buying into the cloud to be hip. Just like Hadoop and other big data technologies they didn’t need in 2014.
Ya just gotta know your business needs.
Ya just gotta know your business needs.
But how many companies are never going to have scaling issues, and also have the resources to hire a good in-house sysadmin to manage their servers? And sure, maybe you have somebody on staff who can handle the sysadmin duties now, but what if they leave? Do you still have the resources to maintain your own servers?
The author's advice is good for basically one scenario: a personal hobby project run by somebody who already knows they would prefer to run their own servers instead of use a cloud host.
The author's advice is good for basically one scenario: a personal hobby project run by somebody who already knows they would prefer to run their own servers instead of use a cloud host.
[deleted]
> But how many companies are never going to have scaling issues, and also have the resources to hire a good in-house sysadmin to manage their servers?
The ones that know how hire good people. If a company can’t hold on to talent then they’ll probably fail eventually. Cloud won’t fix bad management.
The ones that know how hire good people. If a company can’t hold on to talent then they’ll probably fail eventually. Cloud won’t fix bad management.
You need good people before you can hire good people. If you don't already have any sysadmins on your staff, how are you supposed to identify the good sysadmin candidates from the bad ones?
Interviews and recommendations from other talented people in your company. Nothing is gonna be perfect. Not ever company is gonna succeed.
I agree, this guy looks like he's referring to startups with less than 1M views/month. You can get a $20 vps with scaling for that kind of traffic.
> Man I wish HN had a downvote option for articles that are just not worth reading.
I have to strongly disagree with this sentiment. Thoughts which are against mainstream deserve to be visible and thus open for debate.
My company for example decided to go with our own production hardware. I'm not saying it's easy. But it's doable. And very educational. =)
One has to remember that running your systems on 3rd party cloud provider will not strip you from responsibility and/or software maintenance cycle which usually presents most of the "human workload". Their nines are not yours.
According to my experience with two reliable machines (redundant PSU's etc) one can create a very reliable service platform. Using correct kind of management tools of course is essential. RAID disk system and a reliable backup arrangement are essential too.
We used Ganeti (https://en.wikipedia.org/wiki/Ganeti) in our 1st generation setup. The next generation will use our own manager (Deux).
I have to strongly disagree with this sentiment. Thoughts which are against mainstream deserve to be visible and thus open for debate.
My company for example decided to go with our own production hardware. I'm not saying it's easy. But it's doable. And very educational. =)
One has to remember that running your systems on 3rd party cloud provider will not strip you from responsibility and/or software maintenance cycle which usually presents most of the "human workload". Their nines are not yours.
According to my experience with two reliable machines (redundant PSU's etc) one can create a very reliable service platform. Using correct kind of management tools of course is essential. RAID disk system and a reliable backup arrangement are essential too.
We used Ganeti (https://en.wikipedia.org/wiki/Ganeti) in our 1st generation setup. The next generation will use our own manager (Deux).
A $200 DIY hardware cost (equivalent cloud price quoted as $5000 in the article) suggests that, for $4800/mo, you too can do Dev Ops yourself. For a lot of people, that proposition is a hard pass. You want enterprise-grade backups. If your availability drops, your business will lose credibility.
Also, good luck finding a Dev Ops engineer for $57,600 a year.
Also, good luck finding a Dev Ops engineer for $57,600 a year.
Good luck finding a Cloud Engineer for that money to manage your cloud-based infrastructure. ;) Because it’s not going to manage itself, not if you are doing anything remotely complex. And even if you are not doing complex things, DYI is only for techies, cloud or not.
That depends on your definition of 'cloud'. If your app fits into existing PaaS models (Google App Engine, Heroku, et al) then you can dispense with the ops/devops team entirely. Yeah, they're expensive, but nowhere near as expensive as hiring.
You’ll have infrastructure deployment engineers no matter what you do. But you’ll only need infra ops staff (as opposed to application ops staff) if you’re running your own infra.
There’s nobody you pay, in your own company, to sit around watching S3 or Lambda or SQS for degradation. That ops service is part of AWS’s job that you pay for through each service’s fee structure.
There’s nobody you pay, in your own company, to sit around watching S3 or Lambda or SQS for degradation. That ops service is part of AWS’s job that you pay for through each service’s fee structure.
Except, TFA talks about renting a server rather than running your own infrastructure. ;) Sure, with AWS there is less work to do (if you have the skills to set it up in the first place, or it’s £££ time) but TFA is right in that you get much worse performance per £.
Not even really 57,600$. Over here, a person costs their company 1.5x the salary written in their contract (and nearly 2x their "take-home" salary). And most developed countries have something similar (more or less).
So the salary we're really talking about actually is less than 40k a year.
Good luck ^^'
So the salary we're really talking about actually is less than 40k a year.
Good luck ^^'
[deleted]
Comparing it to a full time job is not really fair.
From my experience, doing sysadmin for a single server only takes time initially. You might spend a couple weeks full time to get everything going, but 6 months in it's probably only a couple hours a week.
From my experience, doing sysadmin for a single server only takes time initially. You might spend a couple weeks full time to get everything going, but 6 months in it's probably only a couple hours a week.
I’m glad HN doesn’t have downvoting for articles. It’s probably one of the reasons we still have such a good diverse range of article topics, including controversial ones.
Add downvoting and the article queue would quickly become the same groupthink-swamp that the comments section has become.
One of the nice things about HN is you can still find articles that are against the grain and have topics that challenge the comments section’s “one correct opinion”.
Add downvoting and the article queue would quickly become the same groupthink-swamp that the comments section has become.
One of the nice things about HN is you can still find articles that are against the grain and have topics that challenge the comments section’s “one correct opinion”.
Incredibly ironic that your perfectly useful & reasonable reply is being downvoted.
The internet worked pretty well without Cloud. I reckon a good 90% of the businesses out there don't need cloud, and all that crap about capacity. The issue is that admin cost is very expensive. To get your own network pipe, have someone wire it up, setup your routers, firewall, do backups, etc. That costs tons of money. So even for a small business that can run fine of a raspberry pi, the knowledge to run onprem is so expensive that it's cheaper for them to pay $1000 a month and go to the cloud than pay network/sysadmin $50k a year.
I would bet the vast vast majority of aws customers are not there because of the dynamic resource scaling. Rather, they're there because aws has become the default system provider. At least for the sfbay startup.
I agree, they are very expensive for what you get at any even modest scale. (Except for s3 which is absolutely amazing). However, if you want to run your own stuff, you're buying into knowing the details of how it works. eg you better understand what vacuum is for pg, and how it interacts with availability, and be willing to create backups, etc. AWS does do real work; it's just expensive.
I agree, they are very expensive for what you get at any even modest scale. (Except for s3 which is absolutely amazing). However, if you want to run your own stuff, you're buying into knowing the details of how it works. eg you better understand what vacuum is for pg, and how it interacts with availability, and be willing to create backups, etc. AWS does do real work; it's just expensive.
At one of my previous gigs there was a big push to get out of the colo and into the cloud, part of which because of perceived “cost savings” vs running our own hardware. I think most of it was “everyone else is moving to AWS”.
All I know is we never had a $30,000 “oopsie” in the data center because a dev left a bunch of stuff running they should have shut down.
All I know is we never had a $30,000 “oopsie” in the data center because a dev left a bunch of stuff running they should have shut down.
There are many aspects of our business that are in the cloud today that I am confident do not have scaling challenges, would be just fine running out of a single VPS.
We aren't going to suddenly have 20x employees one day, a lot of our infrastructure is highly predicable and stable in load and demand.
Some things need instant scaling, sure. But not everything that's in the cloud does.
We aren't going to suddenly have 20x employees one day, a lot of our infrastructure is highly predicable and stable in load and demand.
Some things need instant scaling, sure. But not everything that's in the cloud does.
Scaling is not the only thing a cloud offers. And you can get a single VM from any cloud.
RE: >> I'm sorry who is this person<<
https://github.com/jackdoe/txt.black/#start-of-content
https://txt.black/~jack/
In other words this is the OP "zulgan"
user: zulgan created: July 11, 2018 karma: 530 about: https://scrambled-eggs.xyz https://baxx.dev https://github.com/jackdoe
https://github.com/jackdoe/txt.black/#start-of-content
https://txt.black/~jack/
In other words this is the OP "zulgan"
user: zulgan created: July 11, 2018 karma: 530 about: https://scrambled-eggs.xyz https://baxx.dev https://github.com/jackdoe
Their point (in my interpretation) is with bare metal you can get more capacity than you need, for cheaper.
With some cloudflare-ing you could quite easily get away with a single origin server in Germany, too. The latency from a rails app is probably higher anyway.
My anecdata with hetzner - 1.5 decades of lightly used servers for hobby project. One power supply failure resolved in a day. They have some quirks, but.
If you truly have to scale out, sure, cloud is great (sort of). But you can engineer your solution to take the benefit of both, you win.
With some cloudflare-ing you could quite easily get away with a single origin server in Germany, too. The latency from a rails app is probably higher anyway.
My anecdata with hetzner - 1.5 decades of lightly used servers for hobby project. One power supply failure resolved in a day. They have some quirks, but.
If you truly have to scale out, sure, cloud is great (sort of). But you can engineer your solution to take the benefit of both, you win.
Some people buy cars, some people lease them, some people rent them, some people uber.
I think there are valid reasons for each viewpoint.
I think there are valid reasons for each viewpoint.
I'm sorry who is this person and what authority do they have?
His "homepage" is a raw directory dump with a high-contrast black background (the preferred color scheme of 10x engineers).
Isn't that enough authority for you?
BTW if anyone knows of more nuanced explorations of Cloud v. DIY tradeoffs -- we all know there's a tipping point in favor of the latter, somewhere -- please do share.
His "homepage" is a raw directory dump with a high-contrast black background (the preferred color scheme of 10x engineers).
Isn't that enough authority for you?
BTW if anyone knows of more nuanced explorations of Cloud v. DIY tradeoffs -- we all know there's a tipping point in favor of the latter, somewhere -- please do share.
>more nuanced explorations of Cloud v. DIY tradeoffs
100% agree. This article goes so far as to be farcical, but that doesn't mean there isn't something worthwhile to be had in the conversation... as long as that conversation was framed appropriately. Especially around cloud value-adds like managed databases and Lambda and security technologies like CloudWatch/Trail.
I'm the sole (part-time) dev-ops resource for a very small company and we (by which I mean I) DIY just about everything running on a small VPS on DigitalOcean. I run my own database and manage my own dokku and user access etc. I use S3 and have dipped my toes into Lambda but I always wonder if I'm missing out on something not using cloud technologies more heavily.
I would absolutely love an analysis of what cloud technology companies SHOULD be using and at which point they become or are no longer worthwhile in favor of an alternative.
100% agree. This article goes so far as to be farcical, but that doesn't mean there isn't something worthwhile to be had in the conversation... as long as that conversation was framed appropriately. Especially around cloud value-adds like managed databases and Lambda and security technologies like CloudWatch/Trail.
I'm the sole (part-time) dev-ops resource for a very small company and we (by which I mean I) DIY just about everything running on a small VPS on DigitalOcean. I run my own database and manage my own dokku and user access etc. I use S3 and have dipped my toes into Lambda but I always wonder if I'm missing out on something not using cloud technologies more heavily.
I would absolutely love an analysis of what cloud technology companies SHOULD be using and at which point they become or are no longer worthwhile in favor of an alternative.
[deleted]
>Now for a show of hands, how many companies does this describe?
Honestly? Lots.
Honestly? Lots.
Luckily my comment did not end there and I continued with several more rebuttals.
I run my own mini-cloud at home (42u rack with way too much hardware). I do my best to follow best practices with regard to hardware, backups, redundancy, etc. I keep downtime to a minimum (Plex is life, yo). I've been doing it for around 20 years, so I'm fairly confident I know what I'm doing.
As a Director of Technology for a real company, I have never, nor would I ever dream of even considering moving to bare metal for ANYTHING mission critical. The sheer number of possible catastrophies that can occur at the infrastructure level would have me shitting my pants 24/7. Could we hire someone (or a team) to do it? Yes. But EC2 and similar infrastructure beats the pants off those salaries any day.
As a Director of Technology for a real company, I have never, nor would I ever dream of even considering moving to bare metal for ANYTHING mission critical. The sheer number of possible catastrophies that can occur at the infrastructure level would have me shitting my pants 24/7. Could we hire someone (or a team) to do it? Yes. But EC2 and similar infrastructure beats the pants off those salaries any day.
Alongside a host of other issues ("Don't use CDNs because they increase your complexity" is a wild idea coming from someone advocating bare metal boxes) I think there's a confusion of terms when it comes to "cloud"
I prefer renting virtual private servers from Digital Ocean (or, yes, EC2 instances) and configuring them myself. Doing so is perhaps more involved than buying into the AWS/GCP/Azure ecosystem, but I already know the tools involved, and I value the safeguard against vendor lock-in/buy-in enough to spend more time on them.
Those VPSs are "cloud" solutions in my opinion, and a happy medium between AWS and the author's unnecessarily-profanity-laden proposal that everyone should rent/buy metal. You can be "pro-cloud" and "anti-closed-cloud-ecosystem" at the same time.
I prefer renting virtual private servers from Digital Ocean (or, yes, EC2 instances) and configuring them myself. Doing so is perhaps more involved than buying into the AWS/GCP/Azure ecosystem, but I already know the tools involved, and I value the safeguard against vendor lock-in/buy-in enough to spend more time on them.
Those VPSs are "cloud" solutions in my opinion, and a happy medium between AWS and the author's unnecessarily-profanity-laden proposal that everyone should rent/buy metal. You can be "pro-cloud" and "anti-closed-cloud-ecosystem" at the same time.
a big advantage to shoving containers into a larger vps is saving $$ and Mb/s internode.
That's... a point of view.
I'm just glad I can do my things without even having to think about, let alone maintain, 1/10th of all this, really. But YMMV, I guess.
I'm just glad I can do my things without even having to think about, let alone maintain, 1/10th of all this, really. But YMMV, I guess.
This article ends up being a good reason for using cloud. Reads like it was written by "I could write Dropbox in a weekend with rsync" guy.
It's very short sighted to ignore the money being saved by not having to worry about all those things. Their "$5k" example is probably wrong and also irrelevant in most (enterprise) contexts.
Part of me wonders if this is a troll or just a green sysadmin.
Or perhaps just an experienced sysadmin who has different opinions and preferences than you do. There is never a single TheRightWayToDoThings(tm), different people have different needs, priorities and/or resources available.
And an experienced sysadmin would know that there are plenty of needs, priorities, and resources that play well with the cloud. ;)
Respectfully disagree. I'm an ops person, and I have a nostalgic attachment to running my own stuff in boxes that I can touch (or at least visit). However, cloud is just better for most applications. Certainly there are some scenarios where rolling your own is important, but it is uncommon now and only going to increase.
Yes cloud is expensive in a raw performance per $ sense, but that isn't it's value proposition. Most workloads aren't limited by the processing power that you can buy anyway. The draw for cloud is that it abstracts away entire classes of problems. Certainly it provides new problems as well, but those can be worked on in concert with everyone else working on the same platform.
Yes cloud is expensive in a raw performance per $ sense, but that isn't it's value proposition. Most workloads aren't limited by the processing power that you can buy anyway. The draw for cloud is that it abstracts away entire classes of problems. Certainly it provides new problems as well, but those can be worked on in concert with everyone else working on the same platform.
Oh look, all the cloud cheerleaders have come out to mount their defense!
RIP sysadmins, and all the IT knowledge that's slowly disappearing from the web because the sysadmins all work on proprietary stacks at FANG now
RIP sysadmins, and all the IT knowledge that's slowly disappearing from the web because the sysadmins all work on proprietary stacks at FANG now
There was never a better time in the history of the internet where "all the IT knowledge" was more available and easy to get.
And the sysadmins are doing pretty well I am sure, they'll pitch in here once they find their way out of Terraform's documentation.
And the sysadmins are doing pretty well I am sure, they'll pitch in here once they find their way out of Terraform's documentation.
The money you save in the cloud isn’t on hardware, it’s in not having to pay a sysadmin so you can work on business problems.
And that benefit only improves with scale.
And that benefit only improves with scale.
> it’s in not having to pay a sysadmin so you can work on business problems.
...
> that benefit only improves with scale.
That's hard to derive from the same statement - there's a point after which you can afford your own 8 member SRE team across 3 timezones.
The cloud is incredibly attractive when you consider your scale problems as unknown (as a startup, the blitz scale day isn't a good time to go rack nodes yourself) & mostly as way to scale down the costs faster than physical infrastructure + associated human costs can.
I've started looking at this as slightly different model - I'm no longer hiring a good operations manager for those 8 people, I'm hiring a decent programmer who can automate and have code handling deployment tasks instead of a human being taking orders (i.e rack an NVMe + reboot this).
Honestly, even with your physical infrastructure, the trend is towards API based deployment models.
My time at Zynga working on ZCloud, migrating off EC2 to on-prem, that is what I saw. The API based deployment model + reactions from machines (on failure), was worth building everywhere - now that applies to k8s.
That's hard to derive from the same statement - there's a point after which you can afford your own 8 member SRE team across 3 timezones.
The cloud is incredibly attractive when you consider your scale problems as unknown (as a startup, the blitz scale day isn't a good time to go rack nodes yourself) & mostly as way to scale down the costs faster than physical infrastructure + associated human costs can.
I've started looking at this as slightly different model - I'm no longer hiring a good operations manager for those 8 people, I'm hiring a decent programmer who can automate and have code handling deployment tasks instead of a human being taking orders (i.e rack an NVMe + reboot this).
Honestly, even with your physical infrastructure, the trend is towards API based deployment models.
My time at Zynga working on ZCloud, migrating off EC2 to on-prem, that is what I saw. The API based deployment model + reactions from machines (on failure), was worth building everywhere - now that applies to k8s.
Those who think they won't need sysadmins and/or SREs with the cloud are delusional.
I ran a successful marketplace selling and printing >$100MM/yr of tshirts on Google App Engine with no ops or devops. AMA.
*Those who think they won't need sysadmins that are even more expensive
That job doesn't go away, it just changes focus. See DevOps. Generally it is a net positive in my book, but don't believe that operations goes away.
Indeed. Even despite all the shiny web console controls, someone has to know what all the items on the invoice from the cloud provider do, and how they work together.
There more benefit than that. One example...development teams can't force the same level of tech debt that they can on-prem. If Amazon's upgrading some service or competent, there isn't any bargaining or exceptions.
Many similar benefits when you have a huge for-profit entity running the infrastructure at their scale. They optimize a lot for you.
Many similar benefits when you have a huge for-profit entity running the infrastructure at their scale. They optimize a lot for you.
> If Amazon's upgrading some service or competent
AWS almost never forcibly spins down legacy functionality, even if they stop advertising it, unless literally no one is left using it.
That's actually a selling point of AWS to enterprise: while idealistic devs may like “our cloud vendor will force us to use the latest and greatest so we don't have to have that fight”, enterprise buyers don't want “we’re going to be compelled to reengineer our working systems on someone else's timetable”.
AWS almost never forcibly spins down legacy functionality, even if they stop advertising it, unless literally no one is left using it.
That's actually a selling point of AWS to enterprise: while idealistic devs may like “our cloud vendor will force us to use the latest and greatest so we don't have to have that fight”, enterprise buyers don't want “we’re going to be compelled to reengineer our working systems on someone else's timetable”.
RDS might be one example. No forced upgrades from AWS on it? Or AMIs? K8S versions in EKS? Etc.
RDS has user-controlled optional automatic patching and (for SQL Server at least) minor version upgrades. It doesn't have forced upgrades, AFAIK.
But you usually also have to wait longer for new versions of services -- which might not be a problem for the majority, but can be a show-stopper for some.
> The money you save in the cloud isn’t on hardware, it’s in not having to pay a sysadmin so you can work on business problems
Maybe, but you probably have to go pretty far on the cloud maturity spectrum (using cloud native services effectively, not simple VMs like EC2 or even comparatively thin services that abstract just the machine/OS layer like non-Aurora RDS) before you are saving more in local-machine-focused ops teams than you are paying in cloud-focussed ops teams. The latter are more likely to be timeslices of time of team members that are primarily developers, but its still work with a cost.
Maybe, but you probably have to go pretty far on the cloud maturity spectrum (using cloud native services effectively, not simple VMs like EC2 or even comparatively thin services that abstract just the machine/OS layer like non-Aurora RDS) before you are saving more in local-machine-focused ops teams than you are paying in cloud-focussed ops teams. The latter are more likely to be timeslices of time of team members that are primarily developers, but its still work with a cost.
Didn't you see? You're just supposed to learn some sysadmin skills yourself.
The trick is to make yourself the linchpin, so that once the site grows large enough you can never leave, as you're the only one able to keep that writhing mass of wires up and running.
The trick is to make yourself the linchpin, so that once the site grows large enough you can never leave, as you're the only one able to keep that writhing mass of wires up and running.
Yeah, been there, though against my will. Boy was I happy when the system (which actually worked extremely well) was shut down and replaced with a SaaS-based product.
What's different between managing a VPS and an actual server?
who drives to the center to change the disk when it fails
Most data centers provide smart hand service.
You can ask them to do (any) physical task for you.
Virtual disk fails as well.
I actually have had physical boxes with Hetzner, as suggested by the author, for over 10 years, and ran multiple projects on them. Of course I had my share of disk failures. You tell them the disk serial via a special form, tell them when the necessary downtime for the swap is least disturbing for you, and someone switches the disk as requested. 5 minutes later the machine is up again, you trigger RAID rebuild, and that's it.
Doesn't cost a dime to do this, and not more than 20 minutes of work and 10 minutes downtime from my side.
Doesn't cost a dime to do this, and not more than 20 minutes of work and 10 minutes downtime from my side.
"Doesn't cost a dime to do this, and not more than 20 minutes of work and 10 minutes downtime from my side."
Your work isn't free. Application downtime is costing you.
Your work isn't free. Application downtime is costing you.
The exact monetary cost of work time and application downtime (if there even is any, maybe you just shift the load to the second box) varies greatly depending on the context in which the respective project is run (hobby vs. paid-for, I've had both types), which is why I offered time estimates for these costs instead of a monetary amount. The stated monetary amount of "zero" refers to the provider not charging me anything for the new disk itself and the work of changing the disk.
congratulations, you are renting a managed cloud server.
There's common consent that the lowest bar for some hosting to qualify for the "cloud" moniker is that it must abstract from the physical hardware. Most people would even add that it is necessary for some kind of management system to exist which allows for _automated_ provision of your resources.
Renting bare metal instead of owning does not meet this lowest bar. Getting some remote hands service on top with your rental does not change this classification.
Renting bare metal instead of owning does not meet this lowest bar. Getting some remote hands service on top with your rental does not change this classification.
I worry a company will bait and switch with me. I have seen products get discontinued.
But yes, this is outsourcing a problem for a cost.
But yes, this is outsourcing a problem for a cost.
Ugh, what kind of opinion is this? He neglected to mention all the drawbacks:
* It takes forever for Dell or HP to ship your stuff.
* Your co-lo can physically ran out of space.
* Your predecessor will inevitable have tons of one-off unversioned changes in prod.
* Repetitive manual work when scaling multi DCs.
* Making unscalable database choices (non-master-master) because you are not in the cloud mindset.
* Don't worry about CDN? AWS S3 comes with CDN automatically.
* Too many more bullet points to mention.
* It takes forever for Dell or HP to ship your stuff.
* Your co-lo can physically ran out of space.
* Your predecessor will inevitable have tons of one-off unversioned changes in prod.
* Repetitive manual work when scaling multi DCs.
* Making unscalable database choices (non-master-master) because you are not in the cloud mindset.
* Don't worry about CDN? AWS S3 comes with CDN automatically.
* Too many more bullet points to mention.
>Your co-lo can physically ran out of space
The number of times I've run into this at clients is ridiculous. Either there isn't enough space in the rack, or they're out of network switch ports, even down to being out of SFPs or network cables themselves. I've even seen clients run out of power... as in, they couldn't add one of our servers to the data center because the UPS and power delivery systems could not safely handle the load of one more server.
Running a datacenter is a lot harder and more costly than most people think.
The number of times I've run into this at clients is ridiculous. Either there isn't enough space in the rack, or they're out of network switch ports, even down to being out of SFPs or network cables themselves. I've even seen clients run out of power... as in, they couldn't add one of our servers to the data center because the UPS and power delivery systems could not safely handle the load of one more server.
Running a datacenter is a lot harder and more costly than most people think.
> It takes forever for Dell or HP to ship your stuff.
Not a problem for this guy, he uses hetzner's cloud. Did I say cloud, sorry I mean server rental.
Not a problem for this guy, he uses hetzner's cloud. Did I say cloud, sorry I mean server rental.
This guy has a specific problem that's best solved this way. For ten years, I ran my own SQL server, DNS, web servers, game servers, etc. It was the right thing to do. Because I was a hobbyist. Price was everything.
That's not the deal for most startups, though. You're optimizing for the search process. So optimize for the search process. Even after you've found some fit, you're still searching and attempting to grow. Everything that cuts friction from ideation to sale is pure gold. That's why the cloud rules.
I don't figure out how to host the right-sized server with the right infra. I put it on GCS and BigQuery that shit. Or I snowplow into S3 and then Redshift Athena it. Two days to discovering product doesn't work.
If you've ever worked at a big company, you'll know that they have strict IT ops and then a shadow IT operation that manages to be every growth opportunity that gets late merged into the real thing. You want to bypass that. You want to enable people to build things they want to build and sell those things while having automatic best practices. That's the magic of the cloud: instantaneous infrastructure, instantaneous tooling.
$5k is nothing. For the gain I get from that I'd pay out of my own pocket.
That's not the deal for most startups, though. You're optimizing for the search process. So optimize for the search process. Even after you've found some fit, you're still searching and attempting to grow. Everything that cuts friction from ideation to sale is pure gold. That's why the cloud rules.
I don't figure out how to host the right-sized server with the right infra. I put it on GCS and BigQuery that shit. Or I snowplow into S3 and then Redshift Athena it. Two days to discovering product doesn't work.
If you've ever worked at a big company, you'll know that they have strict IT ops and then a shadow IT operation that manages to be every growth opportunity that gets late merged into the real thing. You want to bypass that. You want to enable people to build things they want to build and sell those things while having automatic best practices. That's the magic of the cloud: instantaneous infrastructure, instantaneous tooling.
$5k is nothing. For the gain I get from that I'd pay out of my own pocket.
Part of Friendsters death spiral was the decision to move to AWS, which was a technical disaster.
On Cloud vs baremetal arguments. I respect people find cloud works and is valuable for them. This inspite of my own personal experiences, direct or indirect with cloud have been adverse. I agree with author, for me, for dollar and time effort, baremetal is just better economically and performance.
Most recent. A web service I use heavily migrated from bare metal to cloud and their performance / reliability dropped significantly with numerous outages which tied me up on the phones to my clients. I gave them 6 months to get their act together, as a loyal client then gave up and moved elsewhere. Being a reasonably big client their sales tried to retain then win me back. Conversations were always the same:
Me: Moved because you guys stopped being reliable, reliability is important to me. Ever since you moved to AWS you have been unreliable.
Sales: we had to move to scale and grow
Me: whats the point in moving if moving weakens your product. there are other ways to scale.
Sale: we had to move to scale and grow
On Cloud vs baremetal arguments. I respect people find cloud works and is valuable for them. This inspite of my own personal experiences, direct or indirect with cloud have been adverse. I agree with author, for me, for dollar and time effort, baremetal is just better economically and performance.
Most recent. A web service I use heavily migrated from bare metal to cloud and their performance / reliability dropped significantly with numerous outages which tied me up on the phones to my clients. I gave them 6 months to get their act together, as a loyal client then gave up and moved elsewhere. Being a reasonably big client their sales tried to retain then win me back. Conversations were always the same:
Me: Moved because you guys stopped being reliable, reliability is important to me. Ever since you moved to AWS you have been unreliable.
Sales: we had to move to scale and grow
Me: whats the point in moving if moving weakens your product. there are other ways to scale.
Sale: we had to move to scale and grow
It's not eloquent prose but the message is valid in some ways.
"The Cloud" (presumably AWS/GOOG/AZURE) has become very expensive and very complex.
"The Cloud" (presumably AWS/GOOG/AZURE) has become very expensive and very complex.
Compared to pricing, features and UX cloud has gotten so much cheaper and easier to use if compared to just a few years ago, let alone the first steps AWS took back in the day.
Hard to take seriously and barely rises to the level necessary for rational debate. Is there a workload that is demonstrably 100x slower on AWS or GCE compared to the fully-loaded cost of bare metal?
The TechEmpower benchmarks showed roughly a 10x difference in performance between AWS and bare-metal for basic webserving + DB, across many different languages and frameworks:
https://www.techempower.com/benchmarks/#section=data-r18&hw=...
https://www.techempower.com/benchmarks/#section=data-r18&hw=...
OK but that's a comparison of a $3200 machine with 7x more CPU and more than double the memory of the $100/month virtual machine. The operating costs of the metal machine (electricity, repairs) and its lifetime are unstated.
to be fair techempower benchmarks are not really "world comparable".
it's still just a benchmark. I know a lot of people take these benchmarks as their bread and butter, but there are so many "perf" hacks in them and they do not really have a lot in common with real world usage.
The author was largely focused on specific implementations over architecture patterns, so I wouldn’t think they’re operating at a level where TCO is a consideration...
Well optimized MySQL masters, probably.
"Well optimized MySQL" sounds a bit like having a really attractive liver cancer. Cloud services expand your options and it's worth pondering them. Don't ask what is the best MySQL, ask for a given amount of items I need to store and recall what is the best server OR SERVICE that could do it? MySQL on bare metal? On virtual machines? RDS? Cloud Spanner?
Do cloud providers increase your ability to choose scale-up instead of scale-out? Instead of your sharded and replicated MySQL that is almost inoperable and never seems to maintain consistency, would you be better served with one gigantic database? Sure it costs $13k/month to rent one from Google but it costs a quarter million to buy one from Dell. These are all aspects of the decision-making process.
Do cloud providers increase your ability to choose scale-up instead of scale-out? Instead of your sharded and replicated MySQL that is almost inoperable and never seems to maintain consistency, would you be better served with one gigantic database? Sure it costs $13k/month to rent one from Google but it costs a quarter million to buy one from Dell. These are all aspects of the decision-making process.
MySQL-RDS is the AWS service which I felt gave me the least value for the dollar. (Except for the OpenVPN server which went nuts and did an obscene number of I/Os against EBS which ran up my bill)
As expensive as AWS is, at my company (Fortune 500) company with at least 2 major data centers in the US for my division alone, serving around a hundred clients and thousands of web sites and applications, we found that AWS can still be a lot cheaper than a poorly run private data center. My department is technically a cost center, like most of Security, but the charge-backs we get from our own data center for various services are insanely high compared to the same services we can procure from AWS and Azure. We've been steadily moving our services out of our own data centers and into the cloud and so far, my life is a lot better.
Before, just getting someone to run down a blinking disk light on a server was a chore. Getting them to rack a new server took several days or weeks. Doing anything, any changes, took several days minimum. Now, we can do it all much faster, and from all accounts of my management, it's cheaper.
I don't doubt you can run your own data center cheaper, but not for my company.
Before, just getting someone to run down a blinking disk light on a server was a chore. Getting them to rack a new server took several days or weeks. Doing anything, any changes, took several days minimum. Now, we can do it all much faster, and from all accounts of my management, it's cheaper.
I don't doubt you can run your own data center cheaper, but not for my company.
Back in 2005 I used to work as a systems admin on a relatively high traffic real estate web site. Back at the time, cloud was not a thing, so we had a rented rack in a colo facility and stuffed it full of gear. I remember very well, just how much work and ingenuity it took to keep this thing running, despite the best efforts of our dev team.
He lost it on "don't use CDN". Asia/Europe/America ... if you run a little bit more world wide you will be thankful for every sort of CDN when it's not just text.
This is the author's take on syntax highlighting: https://txt.black/~jack/syntax-highlighting.txt
At least he's consistent.
At least he's consistent.
I dunno. Going from "lift-and-shift" use of EC2 to Cloud Native (lambda, cloudfront, etc.) lowered my AWS bill from $300 a month to $50 a month.
I switched from the dedicated hosting that he advocates to AWS long ago because I was always having things get broken by the people at Softlayer. For instance one time I upgraded my network port and somehow that broke my record in the issue tracking system so I could not longer put issues (or any work reports in.)
There was the clunky and expensive backup server (I'm not sure if it would really restore...) and the breaking point was when somebody added a new disk but they did it wrong so that the partition table got overwritten when the machine rebooted. I was able to fix the partition table but then I moved my files to EC2 as fast as I could because i didn[t have time to deal with that.
As for queues I wonder what is up with that. When I build systems based on message queues they work OK, but I've got an eye for detail and for architecture that many people don't seem to have because I always see other people get in trouble with them.
I switched from the dedicated hosting that he advocates to AWS long ago because I was always having things get broken by the people at Softlayer. For instance one time I upgraded my network port and somehow that broke my record in the issue tracking system so I could not longer put issues (or any work reports in.)
There was the clunky and expensive backup server (I'm not sure if it would really restore...) and the breaking point was when somebody added a new disk but they did it wrong so that the partition table got overwritten when the machine rebooted. I was able to fix the partition table but then I moved my files to EC2 as fast as I could because i didn[t have time to deal with that.
As for queues I wonder what is up with that. When I build systems based on message queues they work OK, but I've got an eye for detail and for architecture that many people don't seem to have because I always see other people get in trouble with them.
Yeah, I think the kicker with managed services vs. self-hosted is whether you're paying for capacity or by usage. The thing with capacity driven costs is that you will always have some unused capacity (and indeed you should as a safety measure). If you're paying by usage, then that pushes the issue of managing overhead capacity onto the service provider.
To be fair, those are cheaper because they want to make you more reliant on AWS and because those are more idyllic for Amazon to run, as with Lyft. Same with AWS Activate and etc.
But for most people, it's simpler to just use them and get it over with, and I'm glad it worked for you.
But for most people, it's simpler to just use them and get it over with, and I'm glad it worked for you.
I disagree with the author but I love reading articles like this.
The core benefit of "the cloud" is missed here. I don't think any serious engineers are saying you CAN'T do it this way, just that nobody wants to anymore.
Over generalization is the main problem here.
It all just depends on the context. Going to the cloud, can really make a lot of sense. For example, if you are a start up that is targeting a mass market and need to scale quickly in multiple regions... Or, if you are the CTO of a large company and don't want to spend time and focus on running your own data center and everything involved with it...
There are also many situations where running your own collocated dedicated server is the best choice. Keeping it lean and mean and removing as much as possible dependencies, complexity and layers does have benefits.
It is hyperbolic to state that something is just shit for everyone in all circumstances.
It all just depends on the context. Going to the cloud, can really make a lot of sense. For example, if you are a start up that is targeting a mass market and need to scale quickly in multiple regions... Or, if you are the CTO of a large company and don't want to spend time and focus on running your own data center and everything involved with it...
There are also many situations where running your own collocated dedicated server is the best choice. Keeping it lean and mean and removing as much as possible dependencies, complexity and layers does have benefits.
It is hyperbolic to state that something is just shit for everyone in all circumstances.
"The cloud (aws, gcloud, azure... whatever) is a piece of shit.
10x price for 1/10th performance (so 100x price haha)"
This ignores the pay of the extra people you need to pay to build/maintain your infrastructure..
This ignores the pay of the extra people you need to pay to build/maintain your infrastructure..
I'm glad this person has an opinion, but it would be nice to know if this opinion is based on how things are today, or two years ago.
Note to bloggers: put dates in your damned posts.
Note to bloggers: put dates in your damned posts.
It seems people are still angry on the internet. These number seem made up and the advice is not realistic. "Avoid big data" What does that even mean?
He means that in his opinion text files are fine for low value analytics until they don’t fit on an SSD you can afford.
In the “roll your own, close to the metal” world this is a pretty sane opinion when the alternative is like, spark or hadoop or something that suck up ops time.
In the “roll your own, close to the metal” world this is a pretty sane opinion when the alternative is like, spark or hadoop or something that suck up ops time.
My summary of this article: Don't go cloud because it will cost you heaps. Instead, spend most of your day sysadmining instead of working on business problems. Or hire a sysadmin.
Sounds like this persons job is being made redundant by clients moving to the cloud and he's not adapting to the market.
Sounds like this persons job is being made redundant by clients moving to the cloud and he's not adapting to the market.
If you manage some legacy systems, as I do, you will find that Cloud is very useful for automation, CI, and dev environments, why? Well, create scripts to build Window servers created with custom images is actually a big deal, so:
- disaster recovery is the first thing you should have in mind, where is the second.
- the right strategy is to preserve the business, not going to the cloud.
I always had fun building my own environments. After the fun came the work and, economically speaking, managing and maintaining those environments has always been an advantage for me. If however, I consider the time it takes to understand and fix problems in VMWare, or SQL optimization, or configure Microsoft AD, Nginx, and others..., I don't know how much I would recommend building everything from scratch by renting some servers.
- disaster recovery is the first thing you should have in mind, where is the second.
- the right strategy is to preserve the business, not going to the cloud.
I always had fun building my own environments. After the fun came the work and, economically speaking, managing and maintaining those environments has always been an advantage for me. If however, I consider the time it takes to understand and fix problems in VMWare, or SQL optimization, or configure Microsoft AD, Nginx, and others..., I don't know how much I would recommend building everything from scratch by renting some servers.
Even for large organizations, the most compelling argument for the cloud is capital structure and the risks associated with holding assets that may not prove to be productive. We inherently understand the advantage of the cloud for startups. It preserves capital and can be used to links expense to generated value (user engagement, etc) if not actual revenue. What may be less obvious is that the same math applies for at least some activities of larger organizations. What goes into a corporate data center is effectively a distressed asset the instant the capacity isn't needed. Forecast risk is very real, and even conservative estimates of that risk can radically change the risk adjusted return on the capital investment.
In many cases, using the cloud is a business and staffing decision more than it is a technical one. It's a decision to accept a monthly cloud bill above a certain threshold instead of maintaining an expensive and perhaps extensive operations staff.
heroku create; git push heroku master; and use my mouse to scale up.
But yeah, sure, the cloud is a piece of shit. Just a beautiful, magical piece of shit.
But yeah, sure, the cloud is a piece of shit. Just a beautiful, magical piece of shit.
I honestly wonder if this is satire. I feel like this is one of those situations where the reason the writer doesn't like it is solely that he doesn't understand the true use case and reasoning behind moving off-prem. It isn't just the cost of servers, it's the fact that AWS or Azure gives you a variety of simple ways to configure and integrate services, without paying an entire team of ops people for putting servers in racks. It's more efficient and reduces overhead, which saves the kind of company that spends 6 figures on AWS costs a huge sum of money too.
Would certainly consider a 'fewer bigger boxes, more hands-on' approach, BUT ONLY for clients that had good administration talent in-house and were already debugging the managed services a lot anyway bc they're pushing the envelope.
Managed services aren't perfect and I've seen them go bad, esp for things like failover. When they hiccup (either perf-wise or downtime) it's hard to debug effectively depending on what logs / stats the vendor makes available.
Managed services aren't perfect and I've seen them go bad, esp for things like failover. When they hiccup (either perf-wise or downtime) it's hard to debug effectively depending on what logs / stats the vendor makes available.
This is good advice overall, in my experience.
Most of the small/medium sized companies I've consulted for could have been easily served by a triple-redundancy Hetzner dedicated server setup for 200 EUR/month. Yet they they chose AWS and paid thousands per month because "that's what everybody does". I have not regularly encountered actual calculation or other analysis as basis for this decision.
Many people here and in industry seem to think that it's either AWS or "build your own datacenter or colocation and maintain machines". Be reminded: This is incorrect. The post recommends renting dedicated servers from a server provider (in this case Hetzner), which includes all hardware handling personnel, hardware replacement, power, and so on, for 50 EUR/month per big-sized server.
Others complain about the "too points in the list". In practice, all these points have to be addressed analogously in an AWS deployment as well, and require approximately as much knowledge as the "sysadmin you'd have to hire". It's a myth that you can just use AWS without reasonable sysadmin knowledge. As a "DevOps engineer" building sophisticated projects on AWS, you usually have to have a superset of typical sysadmin knowledge.
Regarding scalability: For me, Hetzner has reliably fulfilled orders for new dedicated servers within 15 minutes (!) to ssh. In contrast, I've had cases where the AWS support request to raise the limit of 0 big machines in a region to 1 has taken more than 2 business days ("a request of this size has to be escalated").
With a reasonable dedicated server setup, many companies don't need to scale for a while, because you can go a long way with ~5000 SQL inserts that a modern dedicated machine can achieve on an SSD.
Again others highlight the cloud for the available internal bandwidth. Note Hetzner gives you 10 Gbit/s as a 39 EUR/month upgrade. Such bandwidth is reliable; I have measured that I can reliably transfer at Gigabit speed even across the oceans.
A 1 Gbit/s dedicated server with unlimited traffic for 50 EUR/month (at Hetzner) can transfer ~260 TB per month egress. Per AWS calculator, this would cost 18700 USD there -- that's a factor of 375x on traffic cost.
In general, in my experience, the factors you pay for common cloud providers over rented dedicated servers are:
* 10-20x for compute
* 2-4x for storage (this computation includes multi-AZ redundancy)
* ~300x for traffic
The article's point about not being able to use strace/gdb/iostat to debug problems on hosted services is also a good one. For example, we have found AWS EFS to have abysmal performance and could not address this problem. Using CephFS on dedicated infrastructure instead, any bottlenecks could be quickly debugged and performance was great.
Similarly, the point about simplicity is good. Having just simple Linux processes running under systemd on a machine, instead of processes-in-docker-on-kubernetes-in-VMs-on-EC2, removes complexity and makes systems many times simpler and easier to debug, and thus much faster to iterate on.
The only point of the article I don't quite a agree with is to not use CDNs. If you want fast loading times over HTTPs (multiple roundtrips), you need to use a CDN, because of physics. (But you can build a CDN with dedicated servers very easily.)
I have successfully run a triple-redundancy dedicated Hetzner setup over the last 2 years (managed as infrastructure-as-code using NixOS) with a High-Availability database (postgres with Stolon), High-Availability file system (CephFS), systemd and round-robin DNS. So basically, everything the post recommends. It's been the simplest to manage deployment I've used for years, compared to many cloud setups I've worked on. It's had higher uptime than AWS in the last 2 years. It has extremely predictable performance. It costs 200 EUR/month. It's a pleasure for me as a programmer.
In summary, I think the article is on-spot (even though it uses odd language).
There remains one significant benefit of big cloud providers like AWS: Being able to spawn whatever you need via /one/ unified API in very many locations across the world. If you need something that must be distributed across as many data centers as possible, this is quite a plus. But very few people actually need this.
Most of the small/medium sized companies I've consulted for could have been easily served by a triple-redundancy Hetzner dedicated server setup for 200 EUR/month. Yet they they chose AWS and paid thousands per month because "that's what everybody does". I have not regularly encountered actual calculation or other analysis as basis for this decision.
Many people here and in industry seem to think that it's either AWS or "build your own datacenter or colocation and maintain machines". Be reminded: This is incorrect. The post recommends renting dedicated servers from a server provider (in this case Hetzner), which includes all hardware handling personnel, hardware replacement, power, and so on, for 50 EUR/month per big-sized server.
Others complain about the "too points in the list". In practice, all these points have to be addressed analogously in an AWS deployment as well, and require approximately as much knowledge as the "sysadmin you'd have to hire". It's a myth that you can just use AWS without reasonable sysadmin knowledge. As a "DevOps engineer" building sophisticated projects on AWS, you usually have to have a superset of typical sysadmin knowledge.
Regarding scalability: For me, Hetzner has reliably fulfilled orders for new dedicated servers within 15 minutes (!) to ssh. In contrast, I've had cases where the AWS support request to raise the limit of 0 big machines in a region to 1 has taken more than 2 business days ("a request of this size has to be escalated").
With a reasonable dedicated server setup, many companies don't need to scale for a while, because you can go a long way with ~5000 SQL inserts that a modern dedicated machine can achieve on an SSD.
Again others highlight the cloud for the available internal bandwidth. Note Hetzner gives you 10 Gbit/s as a 39 EUR/month upgrade. Such bandwidth is reliable; I have measured that I can reliably transfer at Gigabit speed even across the oceans.
A 1 Gbit/s dedicated server with unlimited traffic for 50 EUR/month (at Hetzner) can transfer ~260 TB per month egress. Per AWS calculator, this would cost 18700 USD there -- that's a factor of 375x on traffic cost.
In general, in my experience, the factors you pay for common cloud providers over rented dedicated servers are:
* 10-20x for compute
* 2-4x for storage (this computation includes multi-AZ redundancy)
* ~300x for traffic
The article's point about not being able to use strace/gdb/iostat to debug problems on hosted services is also a good one. For example, we have found AWS EFS to have abysmal performance and could not address this problem. Using CephFS on dedicated infrastructure instead, any bottlenecks could be quickly debugged and performance was great.
Similarly, the point about simplicity is good. Having just simple Linux processes running under systemd on a machine, instead of processes-in-docker-on-kubernetes-in-VMs-on-EC2, removes complexity and makes systems many times simpler and easier to debug, and thus much faster to iterate on.
The only point of the article I don't quite a agree with is to not use CDNs. If you want fast loading times over HTTPs (multiple roundtrips), you need to use a CDN, because of physics. (But you can build a CDN with dedicated servers very easily.)
I have successfully run a triple-redundancy dedicated Hetzner setup over the last 2 years (managed as infrastructure-as-code using NixOS) with a High-Availability database (postgres with Stolon), High-Availability file system (CephFS), systemd and round-robin DNS. So basically, everything the post recommends. It's been the simplest to manage deployment I've used for years, compared to many cloud setups I've worked on. It's had higher uptime than AWS in the last 2 years. It has extremely predictable performance. It costs 200 EUR/month. It's a pleasure for me as a programmer.
In summary, I think the article is on-spot (even though it uses odd language).
There remains one significant benefit of big cloud providers like AWS: Being able to spawn whatever you need via /one/ unified API in very many locations across the world. If you need something that must be distributed across as many data centers as possible, this is quite a plus. But very few people actually need this.
I really think there's more to this than most people will admit.
Our very early, small startup (4 engineers, low six figures ARR) accumulates $4,000+/month in cloud bills. Its very difficult to manage cloud costs on a small engineering team.
One of our recent examples is with SQS. Our SQS bill is something like $50/month. We process ~100k messages every month. Why is it so expensive, then? It isn't because its processing tons of messages; its because we use open-source polling libraries that are pretty aggressive with their poll rate (then multiply that out by N instances of the poller and there's your bill).
I find it fun to think about "what could I get for $50/month" in regards to a queuing solution. As an extreme counterpoint, Redis running on a raspberry pi in the closet would be a rounding error in our cloud bills over the span of a year. The gray area between these two extremes is where the value lies, and I think that's the author's point: redis on a VPC at Hetzner or DigitalOcean. Billions upon billions of messages processed every month at a fraction of the cost.
So, why pay AWS? There's not NO reason. Reliability is the biggest one I think. But herein lies what I believe to be the intrinsic problem with the cloud: No one can put a price on reliability. If I pay an extra $10k a year to be 99.99999 available (~10s/year), versus 99.99 (~1hr/year), how much extra value does my business receive? No one even CARES what that number is. They just know that more nines is better, so we gotta pay it. But we won't pay too much for it. At some point, it crosses some TOTALLY arbitrary threshold in the CTO's head where its "too expensive" and we need to reduce costs. But, then, we're so bought in that we can't. We can tweak poll rates, we can buy reserved instances, maybe see a 20% reduction, but the pattern is set. We're just a serf to Amazon now.
The other quoted reason is always "the cloud is easier". "You'll save money on HR." "Your engineers can focus on the product." I think Amazon has shouted this line so much from the rooftops that even very smart engineering managers have begun to believe it. Heroku was really the last platform I've used that felt like it could fulfill this promise to any degree of meaning. It really did everything, but AWS has managed to create a network of services that is so fundamentally complex that even when they release services to "simplify" things they actually create more complexity (beanstalk, amplify). You're not getting around having an engineer spending a decent chunk of her time managing this complexity (building CI pipelines, managing IAM permissions, alerting strategies, centralized log ingestion, autoscaling, cost management, encoding everything into CloudFormation, ugh maybe terraform will be easier, figuring out the madness behind VPCs because some random CF stack you found that you thought could automate something for you just spat out an egress-only internet gateway and what the fuck even is that I'm just one person why did I need that).
I've been working in AWS since... the beginning. There are three good AWS services: S3, SQS, and EC2 (in that order). Everything else ranges somewhere between "Fuck it, I guess I'll use cloudwatch" and "what was the rekognition team smoking when they built this, and are they getting a daily dose of it through the vents because its been years and its still not better."
The best part about those "good" services is that they're all solved problems available on a wide variety of cloud providers. Block Storage, Queues, and VMs. I think that's what the author is getting at, in a weird way. The cloud is fine. Managed services on top of open source software is probably fine. Even closed-source managed services can be fine, if they're so stupidly simple that you could explain it to your 8 year old and he'd understand it. But the vast majority of AWS (or Azure, or Google Cloud) is none of these things, and that's the stuff that you should avoid. Or, if you want to save some money, head over to Hetzner or Digital Ocean.
Our very early, small startup (4 engineers, low six figures ARR) accumulates $4,000+/month in cloud bills. Its very difficult to manage cloud costs on a small engineering team.
One of our recent examples is with SQS. Our SQS bill is something like $50/month. We process ~100k messages every month. Why is it so expensive, then? It isn't because its processing tons of messages; its because we use open-source polling libraries that are pretty aggressive with their poll rate (then multiply that out by N instances of the poller and there's your bill).
I find it fun to think about "what could I get for $50/month" in regards to a queuing solution. As an extreme counterpoint, Redis running on a raspberry pi in the closet would be a rounding error in our cloud bills over the span of a year. The gray area between these two extremes is where the value lies, and I think that's the author's point: redis on a VPC at Hetzner or DigitalOcean. Billions upon billions of messages processed every month at a fraction of the cost.
So, why pay AWS? There's not NO reason. Reliability is the biggest one I think. But herein lies what I believe to be the intrinsic problem with the cloud: No one can put a price on reliability. If I pay an extra $10k a year to be 99.99999 available (~10s/year), versus 99.99 (~1hr/year), how much extra value does my business receive? No one even CARES what that number is. They just know that more nines is better, so we gotta pay it. But we won't pay too much for it. At some point, it crosses some TOTALLY arbitrary threshold in the CTO's head where its "too expensive" and we need to reduce costs. But, then, we're so bought in that we can't. We can tweak poll rates, we can buy reserved instances, maybe see a 20% reduction, but the pattern is set. We're just a serf to Amazon now.
The other quoted reason is always "the cloud is easier". "You'll save money on HR." "Your engineers can focus on the product." I think Amazon has shouted this line so much from the rooftops that even very smart engineering managers have begun to believe it. Heroku was really the last platform I've used that felt like it could fulfill this promise to any degree of meaning. It really did everything, but AWS has managed to create a network of services that is so fundamentally complex that even when they release services to "simplify" things they actually create more complexity (beanstalk, amplify). You're not getting around having an engineer spending a decent chunk of her time managing this complexity (building CI pipelines, managing IAM permissions, alerting strategies, centralized log ingestion, autoscaling, cost management, encoding everything into CloudFormation, ugh maybe terraform will be easier, figuring out the madness behind VPCs because some random CF stack you found that you thought could automate something for you just spat out an egress-only internet gateway and what the fuck even is that I'm just one person why did I need that).
I've been working in AWS since... the beginning. There are three good AWS services: S3, SQS, and EC2 (in that order). Everything else ranges somewhere between "Fuck it, I guess I'll use cloudwatch" and "what was the rekognition team smoking when they built this, and are they getting a daily dose of it through the vents because its been years and its still not better."
The best part about those "good" services is that they're all solved problems available on a wide variety of cloud providers. Block Storage, Queues, and VMs. I think that's what the author is getting at, in a weird way. The cloud is fine. Managed services on top of open source software is probably fine. Even closed-source managed services can be fine, if they're so stupidly simple that you could explain it to your 8 year old and he'd understand it. But the vast majority of AWS (or Azure, or Google Cloud) is none of these things, and that's the stuff that you should avoid. Or, if you want to save some money, head over to Hetzner or Digital Ocean.
I really respect the desire and the ability to do it all yourself. I expect that you'll see real gains over the cloud (up until a point) assuming that you're knowledgeable and your environment permits you to work quickly and independently.
https://github.com/jackdoe "old man yells at cloud" - I loled
https://github.com/jackdoe "old man yells at cloud" - I loled
So what's the alternative to not using CDN's for those of us who don't have 100% of our customers in or near Germany?
The reason I run in the cloud to be close to the object storage and average bandwidth between machines.
I run big batch jobs and setting up a full cluster filesystem and fast network is non-trivial.
Recently, using S3 with VMs that have 25Gbit network, I can easily make 'aws s3 cp' (after tuning) generate 10+Gbit of traffic reliably.
I run big batch jobs and setting up a full cluster filesystem and fast network is non-trivial.
Recently, using S3 with VMs that have 25Gbit network, I can easily make 'aws s3 cp' (after tuning) generate 10+Gbit of traffic reliably.
10x price for 1/10th performance (so 100x price haha)
Don't go to the cloud you're such a good sysadmin aha
Don't go to the cloud you're such a good sysadmin aha
As someone mentioned, YMMV. That's all I have to say. Fun side project and learning experience though.
What the cloud gives you (and thus cloud native), is way to manage infra like you manage software objects.
I would say - run everything in kubernetes (cloud or not cloud). Change your infra layer to use kuberentes objects.
Then you minimise the problem to : where is my kubernetes cluster, on cloud, on prem, I do not care.
I would say - run everything in kubernetes (cloud or not cloud). Change your infra layer to use kuberentes objects.
Then you minimise the problem to : where is my kubernetes cluster, on cloud, on prem, I do not care.
So just get a single VM from a cloud provider then. They are all perfectly fine as IaaS if you want to manage it that way, and small VMs are cheap.
This seems to be a massive misunderstanding of what "cloud" even is.
This seems to be a massive misunderstanding of what "cloud" even is.
[deleted]
Does anyone have a good cloud cost calculator that they can recommend?
Web searching shows a few but I'd love some recommendations. Ideally an online spreadsheet, but excel would do too.
Web searching shows a few but I'd love some recommendations. Ideally an online spreadsheet, but excel would do too.
Well I guess cloud is just the new reality now. There's no point fighting windmills really. If it's crappy, but fashionable, it's still fashionable.
I find using the cloud too close to the metal for me. I'm using Netlify for hosting. I get impression they might be using AWS under the hood.
I interviewed with them (didn't get the job). If I understood it correctly, they pretty much use every cloud under the sun to get full coverage for their CDN.
Our times demonstrate that the effective social cost of "the cloud" for sure overtake the provided convenience.
[deleted]
Kind of along the vein of someone telling me I am paying to much for my fancy luxury sports car and completely ignoring the reasons that I find value in it.
"Old man yells at cloud"
Surely this is satire...right?
Haha, did a CMD+F for "satire" as soon as I started reading the comments.
This was the giveaway for me:
This was the giveaway for me:
Don't go to the cloud.
It will force you to... over-complicate your infrastructure to incredible degree.
It is truly a piece of shit and will just force you to design systems in a horrible way."This reads like a parody. Please tell me this is a parody?
Lead times on server orders are non trivial once you are buying a large quantity with specific configurations.
Want to scale up? 5 weeks minimum. Your opportunity may have been lost by then.
Want to scale up? 5 weeks minimum. Your opportunity may have been lost by then.
I use the host he mentioned - they have an API, and my average time to deploy a baremetal physical machine is under 60 seconds via API.
They don't really do custom specs, iirc part of the low pricing is because of how much useful automation they have. Sure, it's not cloud cloud, but if I have a personal side project that I need a few terabytes of ram for, there is no way I can afford AWS/GCP/etc.
They don't really do custom specs, iirc part of the low pricing is because of how much useful automation they have. Sure, it's not cloud cloud, but if I have a personal side project that I need a few terabytes of ram for, there is no way I can afford AWS/GCP/etc.
You can grab new metal and have it all up and running in 60 seconds? That's amazing.
Yes. The API call is here: https://robot.your-server.de/doc/webservice/en.html#get-orde...
The thing is that most baremetal hosts are exactly as the parent described - annoyingly slow, manual, etc.
Hetzner is a little gem here, where they have a large selection of pre-specced machines that are ready for automated deployment, and they also show you a pool of cheaper machines that were returned by other customers/terminated/cancelled and automatically wiped with varying specs that you can also automatically pick out via API (this includes random upgrades that the ex-customers added).
If you manage to fit in the square hole that their automation occupies, it's a great fit.
The thing is that most baremetal hosts are exactly as the parent described - annoyingly slow, manual, etc.
Hetzner is a little gem here, where they have a large selection of pre-specced machines that are ready for automated deployment, and they also show you a pool of cheaper machines that were returned by other customers/terminated/cancelled and automatically wiped with varying specs that you can also automatically pick out via API (this includes random upgrades that the ex-customers added).
If you manage to fit in the square hole that their automation occupies, it's a great fit.
yeah it works if you only need to serve europe. and if you do not need a private network for your servers (for metal they cost a lot of money, even on hetzner, they have a cloud tough which has a network feature). also keep in mind once you try to scale hetzner, is quickly a bottleneck. > 100 servers per hour.
EDIT: oh and also server buying/renting/installing is probably the smallest problem when running your own metal/colo or renting servers. a big use case for clouds is, data sharing between hosts via object storage/moveable disks, etc. which is non trivial on "bare metal/colo/renting" and that is just one use case.
EDIT: oh and also server buying/renting/installing is probably the smallest problem when running your own metal/colo or renting servers. a big use case for clouds is, data sharing between hosts via object storage/moveable disks, etc. which is non trivial on "bare metal/colo/renting" and that is just one use case.
There are certainly a lot of cons, yep. I feel like a lot of the arguments here are a bit apples and oranges, some people are getting really upset over it.
But sometimes, you just need a ton of bandwidth, compute, RAM, etc. and don't care about managed services.. I don't think there's a single major cloud provider that will give me a 2TB Intel DC SSD to use 100% of r/w for $20/m. And not being billed per-GB bandwidth is a massive relief for certain use cases.
But sometimes, you just need a ton of bandwidth, compute, RAM, etc. and don't care about managed services.. I don't think there's a single major cloud provider that will give me a 2TB Intel DC SSD to use 100% of r/w for $20/m. And not being billed per-GB bandwidth is a massive relief for certain use cases.
> I use the host he mentioned - they have an API, and my average time to deploy a baremetal physical machine is under 60 seconds via API.
What makes you think it’s a “baremetal physical machine”? This description sounds very cloud-y to me.
What makes you think it’s a “baremetal physical machine”? This description sounds very cloud-y to me.
I can KVM into it with a Lantronix Spider. I have dozens of them, and none of them have shown to be anything other than a physical machine; I can set up RAID myself, all the hardware data is passed through (there are no hypervisor layers in between).
I would be incredibly impressed if it was not baremetal.
I do have one dislike, which is how harshly they enforce mac addresses (so spinning up new VMs and bridging them into the network gets that mac blocked until you register it in the API or panel). And I guess the other dislike is that they're only in Europe, which is 200ms from me.
I would be incredibly impressed if it was not baremetal.
I do have one dislike, which is how harshly they enforce mac addresses (so spinning up new VMs and bridging them into the network gets that mac blocked until you register it in the API or panel). And I guess the other dislike is that they're only in Europe, which is 200ms from me.
I guarantee you they don’t have the capacity to scale an internet business on demand. Not the way AWS or GCP can.
It's worse than this, having just done it. Trying to be particular about your build? BWAHAHA it's madness.
Said like a person whose never been woken up at 3am due to a hardware issue...
This post is equivalent to that famous comment on Dropbox about running a fileserver yourself. It's a waste of time and I can't believe it's being upvoted on HN.
I would like to have a browser extension that filters all the comments referencing the HN Dropbox meme. It stopped being funny after the 100,000th time.
"Don't go to the cloud, just buy your own servers" completely ignores the reason anyone rents cloud servers in the first place. If I could easily say "I need exactly this much capacity, no more no less, with no unexpected scaling needs and no code/infrastructure changes" then I'd be sitting pretty. Now for a show of hands, how many companies does this describe?
And of those companies that raised their hand, how many can say "I'm fine with just having a server in Germany" who will then go on to say "and I don't need a CDN to serve customers in other regions"?
>keep running live/live setup...This is very critical
Oh I didn't know it was that easy. If all you have to do to keep a server running is say "stay running, it's very critical" then of course no one needs any managed services. Outages are solved forever.
Completely worthless advice from a text file written by... I'm sorry who is this person and what authority do they have?