The Amazon Premium(calpaterson.com)
calpaterson.com
The Amazon Premium
http://calpaterson.com/amazon-premium.html
270 comments
Hi, author here.
The reason I chose the n2-standard-2 over the n1-standard-2 was because the N1 instance in question has only 7.5GB of memory rather than 8GB. I let OVH slip on this one because they didn't have any other instances in the same ballpark but Google did. One of the eggs you have to crack when doing such a comparison I'm afraid.
With respect to the discussion of what really is a fair comparison to 1 unit of Google's proprietary vCPU units: that is of course open to speculation. One of the tremendous problems of cloud computing pricing is that the big three suppliers like to a) take a proprietary unit like "vCPU" which is not in common parlance and b) then repeatedly quote the cost for it in a tiny unit like hours. An n2-standard-4 is 0.0517 USD per hour - great. And let me really get the boot in: I hate that google's pricing page continuously defaults to the this useless unit on every page load. That company is probably the largest producer of web cookies worldwide but to remember that one, essential, UI slider was too much apparently. I broadly file this under the objection of "This is a naive navigation of the pricing schemes of $BIG_TECH" that I cover in the article
Transfer pricing is a real issue and arguably where $BIG_TECH really come and rip your face off. Egress bandwidth is priced at an unreal level precisely because people like me doing comparisons forget about it. In this respect $BIG_TECH are like the chinese restaurants where the dish is £5 but a 250ml bottle of coke to go with it is £6. I should have included it and if I did the comparison again I would model a small defined application rather than a small specific spec size. Mea culpa.
I still think the view I express in the article is right: $BIG_TECH cost a chunk more.
The reason I chose the n2-standard-2 over the n1-standard-2 was because the N1 instance in question has only 7.5GB of memory rather than 8GB. I let OVH slip on this one because they didn't have any other instances in the same ballpark but Google did. One of the eggs you have to crack when doing such a comparison I'm afraid.
With respect to the discussion of what really is a fair comparison to 1 unit of Google's proprietary vCPU units: that is of course open to speculation. One of the tremendous problems of cloud computing pricing is that the big three suppliers like to a) take a proprietary unit like "vCPU" which is not in common parlance and b) then repeatedly quote the cost for it in a tiny unit like hours. An n2-standard-4 is 0.0517 USD per hour - great. And let me really get the boot in: I hate that google's pricing page continuously defaults to the this useless unit on every page load. That company is probably the largest producer of web cookies worldwide but to remember that one, essential, UI slider was too much apparently. I broadly file this under the objection of "This is a naive navigation of the pricing schemes of $BIG_TECH" that I cover in the article
Transfer pricing is a real issue and arguably where $BIG_TECH really come and rip your face off. Egress bandwidth is priced at an unreal level precisely because people like me doing comparisons forget about it. In this respect $BIG_TECH are like the chinese restaurants where the dish is £5 but a 250ml bottle of coke to go with it is £6. I should have included it and if I did the comparison again I would model a small defined application rather than a small specific spec size. Mea culpa.
I still think the view I express in the article is right: $BIG_TECH cost a chunk more.
Given that google tells you what CPUs they use and that a vCPU is one hyper thread on that CPU, it doesn’t seem proprietary at all. Digital Ocean’s shared CPU plans seem like the ones where you’re unsure what you’re buying. Your thesis might still be valid, but you’ve definitely misled readers about Google’s offering. You’re comparing dedicated resources to shared resources when DO and Linode show you that they have higher prices than what you’ve used for equivalent instances.
Edit: also, you can create custom sized instances so you could have created an 8GB 2 CPU N1 instance for $49.63/mo.
Edit: also, you can create custom sized instances so you could have created an 8GB 2 CPU N1 instance for $49.63/mo.
DO also comes with free guaranteed bandwidth.
[deleted]
I'm interested why everyone obsesses over CPU. Everything I've personally ever wanted to do costs more in Bandwidth than CPU. I'm also sure that one of the companies I worked at we spent more on Bandwidth than CPU (until we started doing something super stupid -- caching like several terabytes worth of rendered pages in memory at all times, even though no one ever visited like 90% of the pages over the life of the cache).
Is it the norm for CPU to be a bigger line item than transfer?
Is it the norm for CPU to be a bigger line item than transfer?
> Is it the norm for CPU to be a bigger line item than transfer?
Not universal, but certainly very normal.
1. Transfer within a data center is usually free.
2. CDN reduce transfer fees significantly for static content.
3. Transfer is pay-what-you-use, whereas compute is traditionally pay-what-you-reserve.
AWS transfer are $0/GB incoming; $0.09/GB outgoing. A c5.large (1 physical CPU, 4GB RAM) is $0.09/hour.
So as long as you average less than 291KB/s from your single-core machine, you will spend more on compute than transfer.
Considering that you probably overprovision the CPU to handle spikiness, this is quite likely especially for a typical JSON API server (no images, no video, no web assets).
Not universal, but certainly very normal.
1. Transfer within a data center is usually free.
2. CDN reduce transfer fees significantly for static content.
3. Transfer is pay-what-you-use, whereas compute is traditionally pay-what-you-reserve.
AWS transfer are $0/GB incoming; $0.09/GB outgoing. A c5.large (1 physical CPU, 4GB RAM) is $0.09/hour.
So as long as you average less than 291KB/s from your single-core machine, you will spend more on compute than transfer.
Considering that you probably overprovision the CPU to handle spikiness, this is quite likely especially for a typical JSON API server (no images, no video, no web assets).
I think most customers' biggest spend is on EC2, which is compute. Certainly this has been my experience.
This depends entirely on what your instances are doing. For ardour.org (AWS), we're running a simple PHP/Drupal/Discourse/Mantis web service. Since it's up 24/7 (it's a WEB SERVER!), we have a reserved instance which saves a fair amount on the CPU/compute cost. But even without this saving, data transfer in/out (downloads - customers getting our software; uploads: nightly builds) vastly outweigh the CPU/compute/instance charges.
We picked AWS mostly because they are close to the backbone and because you really get something close to bare metal access. At the time this decision was made, similar access from various other providers was either more expensive or non-existent.
We picked AWS mostly because they are close to the backbone and because you really get something close to bare metal access. At the time this decision was made, similar access from various other providers was either more expensive or non-existent.
Our EC2 is cheap. Our RDS however...
One nitpick on the transfer. DO offers 5TB which at $.12/GB on GCP would be $600 worth of transfer not $8. If you need lots of egress DO wins hands down.
Realistically few are going to need the full 1TB and most will be closer to 2-10% on average (my guess from hosting industry experience).
Realistically few are going to need the full 1TB and most will be closer to 2-10% on average (my guess from hosting industry experience).
In my professional work all of this is someone else's job, but I've experimented with Google Cloud Platform, Amazon Web Services, and Digital Ocean, and only with Digital Ocean did I feel confident I wasn't going to make some kind of costly mistake.
Every time I have to update something on GCP App Engine, I end up creating a billing account so I can use Cloud Build for a Node.js app, then blow the billing account away after the app is deployed. It scares me to create a way to be billed when everything I'm doing is at the free-tier level.
On AWS I discarded my account entirely after I got a bill for a few cents when I wasn't (knowingly) using anything at all.
Every time I have to update something on GCP App Engine, I end up creating a billing account so I can use Cloud Build for a Node.js app, then blow the billing account away after the app is deployed. It scares me to create a way to be billed when everything I'm doing is at the free-tier level.
On AWS I discarded my account entirely after I got a bill for a few cents when I wasn't (knowingly) using anything at all.
AWS charges you for storage, so even if everything is turned off you might have an old snapshot hidden somewhere racking up a few cents a month.
I've found AWS billing to be annoyingly opaque when you're trying to figure out what a service will cost you next month. I guess people who use it a lot have a system, but as a small fry newcomer they seemed really shy about what stuff cost.
I've found AWS billing to be annoyingly opaque when you're trying to figure out what a service will cost you next month. I guess people who use it a lot have a system, but as a small fry newcomer they seemed really shy about what stuff cost.
I often see these comparisons done to try to discredit the use of cloud computing resources, often comparing single cloud entities to bare metal one-for-one comparisons.
But that's now how compute is delivered in projects any more. The value isn't on the box, it's on the platform.
I'm personally about to bolt together a solution that will need to scale significantly, that uses complex message queuing, enormous amounts of object storage, expose set of APIs, and require a highly available distributed document store. And I'm going to personally do it by myself without having to hire 14 SMEs across each of the domains.
The value isn't in a cheaper linux server, or less expensive bandwidth. The value is me being able to develop a solution from the ground up by bolting together Web services, Amazon S3, SNS/SQS, Lambda and DynamoDB from a menu of integrated services all by myself. Period.
But that's now how compute is delivered in projects any more. The value isn't on the box, it's on the platform.
I'm personally about to bolt together a solution that will need to scale significantly, that uses complex message queuing, enormous amounts of object storage, expose set of APIs, and require a highly available distributed document store. And I'm going to personally do it by myself without having to hire 14 SMEs across each of the domains.
The value isn't in a cheaper linux server, or less expensive bandwidth. The value is me being able to develop a solution from the ground up by bolting together Web services, Amazon S3, SNS/SQS, Lambda and DynamoDB from a menu of integrated services all by myself. Period.
> The value isn't in a cheaper linux server, or less expensive bandwidth. The value is me being able to develop a solution from the ground up by bolting together Web services, Amazon S3, SNS/SQS, Lambda and DynamoDB from a menu of integrated services all by myself. Period.
Cool. It sounds like money doesn’t matter to you because you’ve identified a really high margin business and time to market is critical. This doesn’t match how 99% of businesses work though.
Cool. It sounds like money doesn’t matter to you because you’ve identified a really high margin business and time to market is critical. This doesn’t match how 99% of businesses work though.
> Cool. It sounds like money doesn’t matter to you because you’ve identified a really high margin business and time to market is critical. This doesn’t match how 99% of businesses work though.
Cost matters to all businesses. But you have to do proper accounting. Do I hire more people on my team to build automation to create our database clusters, or do I just punch a button and then pay the bill? There must be a point where it makes sense to roll your own, but that point is likely to be fairly high up.
Whenever I hear that someone 'can do it cheaper', I often find that the math is not correct.
By the way, if you are not big enough that the relative cost of a VM between cloud providers is the primary concern, guess what, some of the services mentioned are really cheap – in some cases cheaper than running your own in whatever provider. You don't have to maintain VMs 24/7 to store data on S3, or to make use of SNS, SQS, DynamoDB. Lambda is the poster child for not maintaining servers
If you are really tiny, go use those services instead of wasting any time spinning up servers.
Cost matters to all businesses. But you have to do proper accounting. Do I hire more people on my team to build automation to create our database clusters, or do I just punch a button and then pay the bill? There must be a point where it makes sense to roll your own, but that point is likely to be fairly high up.
Whenever I hear that someone 'can do it cheaper', I often find that the math is not correct.
By the way, if you are not big enough that the relative cost of a VM between cloud providers is the primary concern, guess what, some of the services mentioned are really cheap – in some cases cheaper than running your own in whatever provider. You don't have to maintain VMs 24/7 to store data on S3, or to make use of SNS, SQS, DynamoDB. Lambda is the poster child for not maintaining servers
If you are really tiny, go use those services instead of wasting any time spinning up servers.
I am using Cloud66 that gives me both the advantage. I push code to github and Cloud66 deploy to Digital Ocean. It's very easy to use and I don't need to manage my server at all.
I was using Heroku before, now I am completely satisfied.
PS: If you want to try this is my referral code (10$ off): https://app.cloud66.com/referral?code=IOwgnqPHIEJwEzG
PS: If you want to try this is my referral code (10$ off): https://app.cloud66.com/referral?code=IOwgnqPHIEJwEzG
To most businesses, the cost difference between AWS and barebones services like OVH is negligible. It is a rounding error compared to other operational costs.
I would say the vast majority of SASS where the primary product isn’t some kind of heavy compute/ storage.
Precisely. APIs and ubiquitous services across providers has made multi-cloud possible.
Multi-cloud is a stupid idea. You absolutely should get locked into a single cloud vendor and leverage the shit out of everything they offer.
Somebody else said it best. People go to the cloud not to escape the hardware, it is to escape all the people who administrate the hardware. Truer words have never been spoken. Escaping all the gatekeepers of traditional setups adds significant value to a company.
Somebody else said it best. People go to the cloud not to escape the hardware, it is to escape all the people who administrate the hardware. Truer words have never been spoken. Escaping all the gatekeepers of traditional setups adds significant value to a company.
> Truer words have never been spoken
That might be why uninformed people do it. But the real cost of managing your own hardware is devops, which doesn’t actually go away on the cloud.
Not using multi-cloud or not having a super trivial way to switch is just negligence and wouldn’t be tolerated in any large businesses.
That might be why uninformed people do it. But the real cost of managing your own hardware is devops, which doesn’t actually go away on the cloud.
Not using multi-cloud or not having a super trivial way to switch is just negligence and wouldn’t be tolerated in any large businesses.
I agree. It’s a golden age to be a developer. We can be truly agile with system architecture. The last big system I wrote went through at least three complete architecture overhauls, but all that means is a fun few afternoons on the AWS console. Earlier systems I worked on had to have architecture locked down three months into the project.
I don't think I've ever seen "fun" and "aws console" used in the same sentence before.
The value to me personally is the reliability. Amazon continues to deliver on highly reliable services, which is IMO the main reason they can charge such a premium.
The value to me personally is the reliability. Amazon continues to deliver on highly reliable services, which is IMO the main reason they can charge such a premium.
> I don't think I've ever seen "fun" and "aws console" used in the same sentence before.
In my opinion AWS Console is one of the best designed from all cloud providers out there. GCP is years behind, even with $100k credits I just can force myself to use that pice of unusable crap. Probably Material Design is here to blame, as it's terrible on web.
DigitalOcean does a great job with their interface but they don't even have 1/20 of features offered by AWS, at the end it will share the fate of GCP. Being extremely bloated. :)
In my opinion AWS Console is one of the best designed from all cloud providers out there. GCP is years behind, even with $100k credits I just can force myself to use that pice of unusable crap. Probably Material Design is here to blame, as it's terrible on web.
DigitalOcean does a great job with their interface but they don't even have 1/20 of features offered by AWS, at the end it will share the fate of GCP. Being extremely bloated. :)
My cloud spend is in the millions, soon to be tens of millions, every year and I'm a heavy user of AWS, Google Cloud and Azure and I agree with you wholeheartedly. It's not even just the console but the underlying APIs. I do everything via Terraform anyway but GCP is buggy and inconsistent in areas where it absolutely shouldn't be.
Azure is 5 years ahead of Google Cloud. AWS is 5 years ahead of Azure. The differences are that extreme.
Azure is 5 years ahead of Google Cloud. AWS is 5 years ahead of Azure. The differences are that extreme.
I'm spending literally a fraction of your budget and that is my amateurish assessment as well. AWS Console can be better but the Azure dashboard could've been from the 90s. I had a hard time navigating it.
"In my opinion AWS Console is one of the best designed from all cloud providers out there. GCP is years behind, even with $100k credits I just can force myself to use that pice of unusable crap. Probably Material Design is here to blame, as it's terrible on web."
I hear this for the first time; to me all the Amazon related interfaces I've used have resulted in poor user experience at best. This goes from the store to AWS and sellerscentral should really get a special mention. In AWS, I always felt that everything was super cluttered and important things hard to find, nowhere near intuitive. This goes for high level concepts as much as component behaviors like lists in s3... So, I'd be super curious to hear what you found that was so good. Care to share?
I hear this for the first time; to me all the Amazon related interfaces I've used have resulted in poor user experience at best. This goes from the store to AWS and sellerscentral should really get a special mention. In AWS, I always felt that everything was super cluttered and important things hard to find, nowhere near intuitive. This goes for high level concepts as much as component behaviors like lists in s3... So, I'd be super curious to hear what you found that was so good. Care to share?
I feel this way about azure. The UI is terrible yet I can get stuff done in AWS ui rather quickly.
> I don't think I've ever seen "fun" and "aws console" used in the same sentence before.
To each their own, but the ability to deploy seemingly endless amounts of computing power to do whatever I want, with the click of my mouse, is something I could only dream of as a kid.
To each their own, but the ability to deploy seemingly endless amounts of computing power to do whatever I want, with the click of my mouse, is something I could only dream of as a kid.
I had that kid-like wonder at a recent project where I had to re-architect an Amazon-hosted website for a customer.
It took me back to that childhood experience of playing with Lego. Everything just "snaps together", and a lot of things that ought to be difficult were shockingly easy.
Mid-way through the project the customer apologised because they forgot to mention that they needed geo-redundancy for every layer, including the database.
It was a checkbox on AWS Aurora. A literal checkbox. I pressed it. There was a progress notification and then it was done. Just like that. A distributed cluster.
I know what's involved, it's not like I haven't built SQL Server clusters dozens of times before, I've even scripted the process before. But it's kind of magic to see it condensed down to a the absolutely most trivial true/false button that it could possibly be.
It took me back to that childhood experience of playing with Lego. Everything just "snaps together", and a lot of things that ought to be difficult were shockingly easy.
Mid-way through the project the customer apologised because they forgot to mention that they needed geo-redundancy for every layer, including the database.
It was a checkbox on AWS Aurora. A literal checkbox. I pressed it. There was a progress notification and then it was done. Just like that. A distributed cluster.
I know what's involved, it's not like I haven't built SQL Server clusters dozens of times before, I've even scripted the process before. But it's kind of magic to see it condensed down to a the absolutely most trivial true/false button that it could possibly be.
We might have a different concept of "architecture", but for me it looks like all cloud apps share the very same one. A bunch of (hopefully) cpu bound workers connected to one huge, intransparent persistence service. Not that this is not a viable architecture, but there exist different ones.
You’re not looking beyond the most trivial services. Most complex applications are best designed and scaled through multiple services.
For example, how do you handle long running asynchronous processing, distributed pub sub, or services that have widely different performance and scaling characteristics?
If you haven’t seen this, it might be time to start looking for a more interesting job.
For example, how do you handle long running asynchronous processing, distributed pub sub, or services that have widely different performance and scaling characteristics?
If you haven’t seen this, it might be time to start looking for a more interesting job.
I did not say that you usually need multiple services for scaling. I just noticed that the end architecture looks roughly the same to me once you go into the cloud.
For a truly distributed system just look at a distributed hashtable. You can do it with a cloud provider, but it somehow contradicts the point of a DHT to control the nodes, does it not?
For a truly distributed system just look at a distributed hashtable. You can do it with a cloud provider, but it somehow contradicts the point of a DHT to control the nodes, does it not?
I don’t understand, you acknowledge that vanilla web apps wrapping CRUD aren’t the only architectures but are the only ones suited to cloud? The entire point of the post you’re responding to is that cloud makes many architectures easy, including that one. I can personally vouch for having done wildly different use cases on cloud involving analytics and data processing.
> vanilla web apps wrapping CRUD
This is the only use case you see for what I described? Most data processing I saw works exactly like that, too: some compute cluster using a shared persistence layer (either object store or a network filesystem).
Edge computing, otoh, explicitly exists because not everything can be handled well in that way.
This is the only use case you see for what I described? Most data processing I saw works exactly like that, too: some compute cluster using a shared persistence layer (either object store or a network filesystem).
Edge computing, otoh, explicitly exists because not everything can be handled well in that way.
And to not have to worry about the infrastructure gatekeepers.
Yup. Lots of tin-huggers don't appreciate that it isn't the hardware that people are trying to get away from, it's them.
Speaking of which. My last job I was the dev lead and we were doing everything on prem. Then we hired “consultants” to help us “move to the cloud”. I didn’t know the first thing about AWS when the migration started.
I didn’t know at the time that the consultants were a bunch of old school netops folks who took one AWS exam and all they knew was how to duplicate an on prem infrastructure in AWS.
Of course as always, if you take your on prem thinking and processes to the cloud, you now have the worse of both worlds. All of the process and people overhead of being on prem and you’re spending more for the privilege.
I started studying for the AWS architect associate exam just so I could talk the talk and I realized how much unnecessary work we were doing instead of using managed services. I presented my entire plan to the consultants and they didn’t tell me once that there was a better way.
For phase 2, I wanted to be more “cloud native” there was so much push back from the netops people who could tell they were losing their grip.
Not long after, I changed companies to work for a company where the new CTO - a smart guy who is definitely not young - always prefers managed services.
I’ve worked for small companies and everything I do in AWS, I’ve had to do on prem in the last 20 years. Never again.
I didn’t know at the time that the consultants were a bunch of old school netops folks who took one AWS exam and all they knew was how to duplicate an on prem infrastructure in AWS.
Of course as always, if you take your on prem thinking and processes to the cloud, you now have the worse of both worlds. All of the process and people overhead of being on prem and you’re spending more for the privilege.
I started studying for the AWS architect associate exam just so I could talk the talk and I realized how much unnecessary work we were doing instead of using managed services. I presented my entire plan to the consultants and they didn’t tell me once that there was a better way.
For phase 2, I wanted to be more “cloud native” there was so much push back from the netops people who could tell they were losing their grip.
Not long after, I changed companies to work for a company where the new CTO - a smart guy who is definitely not young - always prefers managed services.
I’ve worked for small companies and everything I do in AWS, I’ve had to do on prem in the last 20 years. Never again.
you don't use cloud to avoid dealing with administrating systems, you use cloud to avoid dealing with systems administrators
ha! you are exactly right. so many gatekeepers in our shops infrastructure teams. a bunch of total clowns who can stall your project for years. all the cloud-haters don't understand why people want the cloud. it is to get rid of them.
Wouldn't the solution for that be to become an infra expert yourself? If you're only buying infra from the cloud aren't you still being gatekept, just in a nicer way?
Becoming an “expert” is just what I’ve been doing since I left that job. I went to a place where the software department guided infrastructure.
Of course as a developer, the thought of creating infrastructure just by writing code is attractive.
Of course as a developer, the thought of creating infrastructure just by writing code is attractive.
[deleted]
> I often see these comparisons done to try to discredit the use of cloud computing resources, often comparing single cloud entities to bare metal one-for-one comparisons.
The post is not about cloud computing vs bare metal, it is about AWS compute and storage being more expensive. All the alternatives presented are cloud providers. The article provides data, you are providing hyperbole. You are not even wrong. AWS probably IS the most feature complete PAAS, but you can't deny it's also the most expensive IAAS.
The post is not about cloud computing vs bare metal, it is about AWS compute and storage being more expensive. All the alternatives presented are cloud providers. The article provides data, you are providing hyperbole. You are not even wrong. AWS probably IS the most feature complete PAAS, but you can't deny it's also the most expensive IAAS.
I still think this misses the point. Looking at individual services (especially the most comparable "commoditized" services like EC2 and S3) is not how most people derive value from cloud services these days. In fact, as someone who just made a cloud decision for a company, pricing of individual services almost wasn't even on my radar. At the end of the day these differences are almost inconsequential to me compared to bigger issues like "is this service missing critical features". For example, I really like Google's cloud, but I'm a bit baffled their postgres Cloud SQL offering still doesn't offer point-in-time-recovery, which is a critical need for me, while AWS has had it for a long time. So do you really think I'm worried that S3 buckets may be slightly more expensive, compared to how much more it would cost me in developer time if I had to manage my own postgres instances on a compute machine?
I do think the data in this article was valuable, and it probably is likely that AWS charges more for these services because they can - the total value people get from the whole AWS system makes it worth it. So if you only care about a compute instance and storage, you probably don't need AWS, but the second the decision becomes "Do I have to hire a developer if I have to do this thing on my own?" it becomes pretty clear why people want a more full-featured cloud.
I do think the data in this article was valuable, and it probably is likely that AWS charges more for these services because they can - the total value people get from the whole AWS system makes it worth it. So if you only care about a compute instance and storage, you probably don't need AWS, but the second the decision becomes "Do I have to hire a developer if I have to do this thing on my own?" it becomes pretty clear why people want a more full-featured cloud.
I'm not sure it has to stay this way though. I think turn-key private cloud is almost within reach. We clearly aren't there now, but it'll come in time.
Get to a certain maturity and standardization of the cloud software. Don't deliver servers, deliver half-racks, pre-assembled, Turn-key. No screwdrivers, no wiring to braid, just a hundred cores on a couple dozen motherboards and a couple 10G wires coming out the side.
Get to a certain maturity and standardization of the cloud software. Don't deliver servers, deliver half-racks, pre-assembled, Turn-key. No screwdrivers, no wiring to braid, just a hundred cores on a couple dozen motherboards and a couple 10G wires coming out the side.
There's really no controversy that AWS, Azure, and GCP are substantially more expensive than bare metal providers, if you're simply comparing machine specs.
So if all you're doing is setting up root access and handling everything yourself, then yes, you're throwing huge wads of cash down the toilet with those platforms versus renting a VPS on Linode, Digital Ocean, etc.
The economic disparity can flip, though, if you use these services as they're intended. i.e., take advantage of features that are charged incrementally and optimise to reduce how much they're activated per user transaction, where "user transaction" is a key user action such as a page serve or a reported being emailed.
The costs can be much lower when you're only using what you need, and not paying for large capacity to be sitting around in case of a peak usage moment. And even more so when you consider there are automatic upgrades and ops to ensure high availability.
So if all you're doing is setting up root access and handling everything yourself, then yes, you're throwing huge wads of cash down the toilet with those platforms versus renting a VPS on Linode, Digital Ocean, etc.
The economic disparity can flip, though, if you use these services as they're intended. i.e., take advantage of features that are charged incrementally and optimise to reduce how much they're activated per user transaction, where "user transaction" is a key user action such as a page serve or a reported being emailed.
The costs can be much lower when you're only using what you need, and not paying for large capacity to be sitting around in case of a peak usage moment. And even more so when you consider there are automatic upgrades and ops to ensure high availability.
Easier said than done. To implement that I will probably need to hire a DevOps guy, and now we have all this cloud formation (or whatever your infra as code choice is) code to manage now. So in reality it probably costs more (devops and more code to manage) than if I just went with a couple cheaper bare metal servers.
If you are running in the cloud then you still need a devops guy same as you would if you where bare metal. In fact you will probably need _more_ devops people the deeper you get into the AWS ecosystem.
If you are running in the cloud then you still need a devops guy same as you would if you where bare metal. In fact you will probably need _more_ devops people the deeper you get into the AWS ecosystem.
I disagree. Any of our senior developers can create a CloudFormation template or use one that we already have and make minor changes and include in their repository.
Every place that I have worked it’s the responsibility of the team who wrote the code to create the CI/CD pipelines.
Every place that I have worked it’s the responsibility of the team who wrote the code to create the CI/CD pipelines.
Wait, are you really practicing an environment where your developers have an understanding of operations.
Dang, I wish we had a term for this blending of roles...
Dang, I wish we had a term for this blending of roles...
We have grown in complexity since we first started to have a dedicated “ops team”, but honestly it’s because developers just didn’t want to do the grunt work and we needed someone to make sure everything was done consistently.
But still ops serve developers not the other way around. The senior developers who knew AWS well, basically set the standards and kept ourselves accountable to the ops guy we hired, even though any of us can override him because of our influence in the company.
I started taking away some of my own access and privileges just so I would be the first to hit roadblocks to feel other developers pains who weren’t given the keys to kingdom.
But still ops serve developers not the other way around. The senior developers who knew AWS well, basically set the standards and kept ourselves accountable to the ops guy we hired, even though any of us can override him because of our influence in the company.
I started taking away some of my own access and privileges just so I would be the first to hit roadblocks to feel other developers pains who weren’t given the keys to kingdom.
But you understand Ops, and you have your developers understand Ops, which is my point.
Hiring "DevOps" teams completely misses the point, in the same way that I don't hire Unit Testing teams to write the unit tests that my Devs don't want to do the grunt work for.
When a Dev understands Ops they write more efficient code, as they realise what storing your entire DB in cache really means for the server.
Hiring "DevOps" teams completely misses the point, in the same way that I don't hire Unit Testing teams to write the unit tests that my Devs don't want to do the grunt work for.
When a Dev understands Ops they write more efficient code, as they realise what storing your entire DB in cache really means for the server.
This is my experience too, albeit sometimes it does feel good to have an infra specialist on the team. Edge cases do happen.
That’s what the AWS Business Support is for.
But at least three of the senior engineers (including me) I feel could hold our own against any “specialists”. My experience are too many of the “specialists” are old school netops people who got one certification and treat AWS like an overpriced colo.
But at least three of the senior engineers (including me) I feel could hold our own against any “specialists”. My experience are too many of the “specialists” are old school netops people who got one certification and treat AWS like an overpriced colo.
AWS likes to pawn their customers off to Certified Partners for outsourced solutions.
I'm a software engineer who went the specialist route because it does take real skill to do this well. Yes, I am embedded on a team of old school netops people now, but I'm in charge of all of this and I get to drag them kicking and screaming in to the modern world.
Specialists are worth it if you find the right one.
I'm a software engineer who went the specialist route because it does take real skill to do this well. Yes, I am embedded on a team of old school netops people now, but I'm in charge of all of this and I get to drag them kicking and screaming in to the modern world.
Specialists are worth it if you find the right one.
I’m a software developer/architect/team lead/single responsible individual depending on how the wind blows, but after a few years of adding AWS to my toolbelt, I think I can hold my own and I have been recruited to be on the infrastructure side.
Old school netops folks are so afraid of becoming less relevant they do their best to keep control. But at least they are harmless compared to the ones that have tried to transition to the cloud. They are actively harmful costing clients and companies more with little to show for it.
And no I am not young. I’m 45 and started programming in assembly in the 80s.
“lift and shift” should be phase 1. Not the end goal.
Old school netops folks are so afraid of becoming less relevant they do their best to keep control. But at least they are harmless compared to the ones that have tried to transition to the cloud. They are actively harmful costing clients and companies more with little to show for it.
And no I am not young. I’m 45 and started programming in assembly in the 80s.
“lift and shift” should be phase 1. Not the end goal.
I think we pretty much see eye to eye here :)
> Easier said than done. To implement that I will probably need to hire a DevOps guy, and now we have all this cloud formation (or whatever your infra as code choice is) code to manage now. So in reality it probably costs more (devops and more code to manage) than if I just went with a couple cheaper bare metal servers.
This does not make any sense. You don't need cloudformation or anything, you can just use a wizard and provision as many VMs (baremetal or otherwise) as you need. It's literally a form and next -> next -> next.
Now that you have systems you can login to, their complexity is the same – except you won't have to care, or manage any hardware.
You still have to manage those systems yourself. Keep them patched, secure, and the workloads up. It's your choice whether or not to delve deeper into the AWS ecosystem.
Note that even though I said you don't need cloudformation (actually just use Terraform instead), you have a lot of power at your disposal if you do. You can't automate racking and stacking of physical servers, but you can fully automate the lifecycle of a cluster of VMs. At my job I can bring up a 40+ cluster containing many kinds of workloads, with a single butotn press. And destroy it just as easily (for non-prod). That's invaluable.
This does not make any sense. You don't need cloudformation or anything, you can just use a wizard and provision as many VMs (baremetal or otherwise) as you need. It's literally a form and next -> next -> next.
Now that you have systems you can login to, their complexity is the same – except you won't have to care, or manage any hardware.
You still have to manage those systems yourself. Keep them patched, secure, and the workloads up. It's your choice whether or not to delve deeper into the AWS ecosystem.
Note that even though I said you don't need cloudformation (actually just use Terraform instead), you have a lot of power at your disposal if you do. You can't automate racking and stacking of physical servers, but you can fully automate the lifecycle of a cluster of VMs. At my job I can bring up a 40+ cluster containing many kinds of workloads, with a single butotn press. And destroy it just as easily (for non-prod). That's invaluable.
> Now that you have systems you can login to, their complexity is the same – except you won't have to care, or manage any hardware.
Depends upon the number of servers you have.
Back when I worked on several somewhat popular websites ( a handful with ~1-5mil daily unique users), we had about 40 servers and they mostly took care of themselves. Between me (primarily a developer) and the CTO we averaged maybe a single day thinking about hardware per month, and that was mostly to install new hardware rather than taking care of existing stuff.
If you have this number of servers, once you have something like Ansible setup (we used cfengine back in the day, ugh), both hardware and software mostly manages itself.
Depends upon the number of servers you have.
Back when I worked on several somewhat popular websites ( a handful with ~1-5mil daily unique users), we had about 40 servers and they mostly took care of themselves. Between me (primarily a developer) and the CTO we averaged maybe a single day thinking about hardware per month, and that was mostly to install new hardware rather than taking care of existing stuff.
If you have this number of servers, once you have something like Ansible setup (we used cfengine back in the day, ugh), both hardware and software mostly manages itself.
in 2007/2008, I managed a GIS server / website that averaged 100 hits per day, but once every couple years the website would get listed in time/cnn and would get millions of hits.
I set up an EC2 instance behind a load balancer and set it to auto-scale. done. If I had to handle that bare bones, I would have had to upgrades switches, manage dozens of servers, deal with hard drive failures, etc., and most of that would be idle 99% of the time.
I set up an EC2 instance behind a load balancer and set it to auto-scale. done. If I had to handle that bare bones, I would have had to upgrades switches, manage dozens of servers, deal with hard drive failures, etc., and most of that would be idle 99% of the time.
As someone who is often categorized as a devops guy, there really shouldn’t be such a role. The whole point of devops is that you have developers that can perform operations tasks.
Devops means different things to different people. I'm a devops guy surrounded by devops guys and we all are operations people that develop, which day to day is completely different from a developer that performs operations.
What are you comparing? If you’re comparing a large, HA SaaS use case with a static website on a VPS, then of course the latter requires less DevOps work, but if you hold the use cases equivalent, AWS requires much less DevOps work than bare metal. Notably, just because you’re using bare metal doesn’t mean the motivation for infra-as-code goes away; to the contrary, you need more of it because you’re now managing more services that come out-of-the box on AWS.
Your static site would still probably be better on AWS. Whack it into an S3 bucket, put cloudfront in front of it, and you have a globally scalable, CDN-enabled static site with at least five 9's of reliability and it costs you peanuts.
>> and now we have all this cloud formation ... code to manage
The cloud formation code is (likely to be) much less than your application code. ... and if your intention and need is to have couple servers, then you don't really need any infrastructure code. If these cases, yes bare metal is much cheaper and (probably) better option.
The cloud formation code is (likely to be) much less than your application code. ... and if your intention and need is to have couple servers, then you don't really need any infrastructure code. If these cases, yes bare metal is much cheaper and (probably) better option.
Datapoint: We have 2 "DevOps guys" supporting a significant AWS infrastructure. We autoscale from 200 ec2 instances at night to 700 ec2 instances during the day. We run 60+ microservices, each of which has multiple processes that run, each of which is autoscaled (we use ECS). We use Aurora (with autoscaled readers) and DynamoDB (autoscaled IOPS). We manage all of that with 2 "Devops Guys".
Granted, we're a mature startup and have put a few years of investment (at the cost of 2-3 "Devops Guys") into our infra, but ultimately it doesn't take much to manage a ton of AWS infra once the tooling is in place.
Granted, we're a mature startup and have put a few years of investment (at the cost of 2-3 "Devops Guys") into our infra, but ultimately it doesn't take much to manage a ton of AWS infra once the tooling is in place.
Man, that sounds so luxurious. I'm begging for us to hire a second guy because I'd like to not always be on point for everything and to take vacations. Probably running an order of magnitude more stuff than you described, multi-cloud and with Terraform.
Terraform and the fact that I came in with experience makes this doable. But only just.
Terraform and the fact that I came in with experience makes this doable. But only just.
[deleted]
just for back of the envelope, who many customers are you able to support per ec2 instance?
WhatsApp used to be hosted on ~15 bare metal servers serving 100 million concurrent users...
They were also acquired at a price which would value each employee at ~350M.
They were capable of scaling in a way that is certainly an anomaly, and not indicative of the costs of an ordinary team.
It speaks volumes about what the right talent and architecture/technology choices can do if leveraged successfully, but is more of an interesting anecdote than a realistic infrastructure budget.
They were capable of scaling in a way that is certainly an anomaly, and not indicative of the costs of an ordinary team.
It speaks volumes about what the right talent and architecture/technology choices can do if leveraged successfully, but is more of an interesting anecdote than a realistic infrastructure budget.
> They were also acquired at a price which would value each employee at ~350M.
That’s a pointless calculation. The acquisition wasn’t for the employees. As with all network-effects products, the acquisition was for the active user base. They could have acquired WhatsApp, fired the engineering team, rewrote it with an architecture that required 100x the servers and still been happy.
That’s a pointless calculation. The acquisition wasn’t for the employees. As with all network-effects products, the acquisition was for the active user base. They could have acquired WhatsApp, fired the engineering team, rewrote it with an architecture that required 100x the servers and still been happy.
It speaks volumes about Erlang/BEAM I think
I find this so fascinating, is there more info on the software/hardware during this time period?
Before ist was possible to Share Images. BTW they used s3 dir that
We have give or take 30M monthly active users.
The instances we use are not the largest (we use 2xls) but we also incorporate spot instances as part of our autoscaling.
The instances we use are not the largest (we use 2xls) but we also incorporate spot instances as part of our autoscaling.
[deleted]
[deleted]
What do you need if you're managing a significant amount of metal? No devops guys?
Everything is a tradeoff.
You can manage things efficiently in AWS if you do it right. You can manage metal efficiently if you do it right.
You can kill your business if you do either one wrong.
Everything is a tradeoff.
You can manage things efficiently in AWS if you do it right. You can manage metal efficiently if you do it right.
You can kill your business if you do either one wrong.
One of the benefits of the cloud is that the developers should be able to easily manage their own infrastructure. After all, they should be the people most familiar with the performance profile of their service/micro service/application. They should be the ones making decisions like using Aurora vs Dynamo vs managing your own dB on vms or bare metal vs a Hadoop cluster across VMs. They should own their deployment pipeline with CI/CD. If you have a dedicated DevOps person or team on a pure cloud application you are either a very large organization that is coordinating across multiple development teams that each have their own infra. Or you have built something brittle and not entirely cloud native (eg self managed Cassandra or elastisearch on a cluster of VMs). (Third possibility is a complex micro service architecture where it’s nice to have someone purely in-charge of “the system view” of the infrastructure even with a small number of developers.)
Sounds like a good way to get a runaway bill.
Why? Do You think developers can’t consider cost/performance? Do you think engineering managers and their finance partners don’t care? Maybe I’ve gotten lucky with company choice, but the engineer who finds massive cost savings due to optimization gets recognized over someone implementing the next basic feature.
Are you saying that the developers are unable to look at a bill?
In that case you can use Heroku or a 'Managed Cloud Experts' like Rackspace
I don't know about AWS, but it's also easier for us to sell to clients by saying that Azure takes care of the database updates and backups for us, and can be incredibly robust if they require, and everything is available at the flick of a switch. They are paying a premium for the convenience, but it's a lot of convenience.
Yeah, when I had my own company “everything is on Azure” was a generally satisfying response to “where the heck do you keep my data.”
This is a huge factor. A lotta clients _really want_ to hear "AWS" or "Azure" when they ask "..and where is the infrastructure hosted"
That's absolutely true if you're selling to businesses (especially large businesses).
I had the same experience hosting on AWS (having previously hosted on colocated bare metal server). An entire category of sales friction almost entirely disappeared, and it was friction that was common on the largest and most profitable accounts.
I had the same experience hosting on AWS (having previously hosted on colocated bare metal server). An entire category of sales friction almost entirely disappeared, and it was friction that was common on the largest and most profitable accounts.
The thing is, he's comparing virtual machines at all of these providers.
If you want to use managed services (Fargate, Aurora, DynamoDB, etc) at AWS, it's even more expensive!
So if you just use EC2 then alternatives are cheaper, and if you use managed services it becomes so expensive so can actually afford hiring people to take care of your virtual machines.
If you want to use managed services (Fargate, Aurora, DynamoDB, etc) at AWS, it's even more expensive!
So if you just use EC2 then alternatives are cheaper, and if you use managed services it becomes so expensive so can actually afford hiring people to take care of your virtual machines.
If you account for engineering effort, Fargate is cheaper than EC2. No central logging or log exfiltration to configure, no bin packing apps onto AMIs, no SSH to set up or keys to manage, no configuration management, no process management to worry about, etc.
Just put your app into an image and write the CF templates for your cluster, services, and task definition and you’re good to go.
Just put your app into an image and write the CF templates for your cluster, services, and task definition and you’re good to go.
Are you including HA and maintenance? A typical network engineer where we live has a fully allocated cost of around $170K. That can pay for a lot of stuff on AWS.
And we can actually move faster without the netops bottle neck.
And we can actually move faster without the netops bottle neck.
Oh boy, that canard. In my experience, cloud management is nothing like as simple as people make out; and you're not going to be magically making those ifrastructure man-power costs just go away. You need to configure all that stuff non-the-less, and that's where the time goes. Even compared to bare-metal, the difference isn't all it's chalked up to be - the actual rusty box doesn't need a lot of TLC, and you can hire companies for that too, and they just don't break all that often for this to matter much. I mean; who wants the hassle; and it's still sane to outsource stuff you're not good at and don't (want to) care about - but it's not a huge source of savings.
And HA is - for most usages - best avoided. It's a complexity source like nothing you've seen before. Better to make sure you can replace anything that's broken really fast (because it's all scripted), and if you have enough scale to have at least few machines - just accept that you'll have some very small downtime when lightning strikes, and that many machines can be temporarily missed without much impact anyhow. By the way - you'll have that in HA cloud datacenters too, but the cause there will be human error due to the excessive complexity. If you dive into uptime claims by cloud providers it's typically for obtuse stuff like "can bare VM ping another VM over http in same datacenter". Actually use any of all of those complex services they provide, and weird stuff like "the outside internet connectivity" and the uptime of the overall system starts shedding those precious 9's. Uptime is almost certainly excellent compared to any trivial alternative, but the real-world difference isn't as large as the misleading uptime claims lead you to believe. I've had outages cloud providers several times while their uptime ticker claimed everything was just peachy.
If you really need HA, my condolences - but it's overrated vs. simply being able to spin up a new copy in a few minutes, especially since stuff just doesn't die or crash all that often; typically only once every few years, if that. Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.
And HA is - for most usages - best avoided. It's a complexity source like nothing you've seen before. Better to make sure you can replace anything that's broken really fast (because it's all scripted), and if you have enough scale to have at least few machines - just accept that you'll have some very small downtime when lightning strikes, and that many machines can be temporarily missed without much impact anyhow. By the way - you'll have that in HA cloud datacenters too, but the cause there will be human error due to the excessive complexity. If you dive into uptime claims by cloud providers it's typically for obtuse stuff like "can bare VM ping another VM over http in same datacenter". Actually use any of all of those complex services they provide, and weird stuff like "the outside internet connectivity" and the uptime of the overall system starts shedding those precious 9's. Uptime is almost certainly excellent compared to any trivial alternative, but the real-world difference isn't as large as the misleading uptime claims lead you to believe. I've had outages cloud providers several times while their uptime ticker claimed everything was just peachy.
If you really need HA, my condolences - but it's overrated vs. simply being able to spin up a new copy in a few minutes, especially since stuff just doesn't die or crash all that often; typically only once every few years, if that. Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.
* In my experience, cloud management is nothing like as simple as people make out; and you're not going to be magically making those ifrastructure man-power costs just go away. You need to configure all that stuff non-the-less, and that's where the time goes.*
I wrote CloudFormation scripts that handle our basic use cases and use parameters.
Any developer can click on quick create link for instance and have an isolated build environment just for there service. If they need something above and beyond the standard CodeBuild Docker container for their build environment they can create their own.
And we have made the infrastructure manpower diminish to an extent that it might as well be gone.
* Even compared to bare-metal, the difference isn't all it's chalked up to be - the actual rusty box doesn't need a lot of TLC, and you can hire companies for that too,*
Can I hire a company to put in 20 additional servers every night when our processing queue increases and take them out when it decreases?
And HA is - for most usages - best avoided. It's a complexity source like nothing you've seen before
It’s “complex” to have an autoscaling group with a min/max of two and clicking on a button (well actually use a CF script) when you want to increase the number of servers? There is nothing “complex” about having HA in a cloud environment.
Better to make sure you can replace anything that's broken really fast (because it's all scripted),
It’s even faster when you don’t have to do it yourself..
* Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.*
HA can be as simple as a rogue query killed an app/web server (been there done that). I got an alert that the web server was unhealthy. By the time I woke up well, I got another alert saying that a new instance was brought up. I went back to sleep and investigated in the morning. Not to mention the automatic updates performed on RDS instances or the updates that you don’t have to care about with managed services.
I wrote CloudFormation scripts that handle our basic use cases and use parameters.
Any developer can click on quick create link for instance and have an isolated build environment just for there service. If they need something above and beyond the standard CodeBuild Docker container for their build environment they can create their own.
And we have made the infrastructure manpower diminish to an extent that it might as well be gone.
* Even compared to bare-metal, the difference isn't all it's chalked up to be - the actual rusty box doesn't need a lot of TLC, and you can hire companies for that too,*
Can I hire a company to put in 20 additional servers every night when our processing queue increases and take them out when it decreases?
And HA is - for most usages - best avoided. It's a complexity source like nothing you've seen before
It’s “complex” to have an autoscaling group with a min/max of two and clicking on a button (well actually use a CF script) when you want to increase the number of servers? There is nothing “complex” about having HA in a cloud environment.
Better to make sure you can replace anything that's broken really fast (because it's all scripted),
It’s even faster when you don’t have to do it yourself..
* Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.*
HA can be as simple as a rogue query killed an app/web server (been there done that). I got an alert that the web server was unhealthy. By the time I woke up well, I got another alert saying that a new instance was brought up. I went back to sleep and investigated in the morning. Not to mention the automatic updates performed on RDS instances or the updates that you don’t have to care about with managed services.
> > In my experience, cloud management is nothing like as simple as people make out; and you're not going to be magically making those ifrastructure man-power costs just go away. You need to configure all that stuff non-the-less, and that's where the time goes.*
> I wrote CloudFormation scripts that handle our basic use cases and use parameters.
scripts will run on things other than the cloud. The thing that made that difference? It may have been you, not the cloud.
> > Even compared to bare-metal, the difference isn't all it's chalked up to be - the actual rusty box doesn't need a lot of TLC, and you can hire companies for that too,*
> Can I hire a company to put in 20 additional servers every night when our processing queue increases and take them out when it decreases?
Nope; you can't - and that's a real upside to a cloud. I'm not opposed to using the cloud; merely saying that personnel costs savings are nonsense, however. In any case; you cannot easily do this with physical hosts. Then again; most people probably don't need to. If your workload is merely a little spiky (like day vs. night) then the extra costs due to the cloud will be greater even if you spin down instances sometimes; and if you do that, you're spending time to do so, which undercuts the clouds other selling point "I don't want to manager all that junk".
On HA: You're listing scaling, not HA. You don't need to do anything special to make interchangable, largely stateless machines "HA" no matter your infrastructure. HA is relevant if you have any single points of failure, and how you avoid the consequences of having those; e.g. having a live database replica or whatever. But sure: this 'll be easier on some clouds!
> > Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.
> HA can be as simple as a rogue query killed an app/web server (been there done that). I got an alert that the web server was unhealthy. By the time I woke up well, I got another alert saying that a new instance was brought up. I went back to sleep and investigated in the morning. Not to mention the automatic updates performed on RDS instances or the updates that you don’t have to care about with managed services.
So... why does any of that need the cloud? Sounds like a plain old process or container needed restarting. That's just basic server stuff servers have been doing for... well, longer than the cloud even existed.
But the point isn't that the cloud is a bad idea. It's that people ascribe frankly absurd benefits to it, when many of those are either unrelated to being in a cloud, and some are wildly overstated, specifically the management costs. Not that there aren't upsides: but if you're saving 170k on network engineering personnel costs; well, you didn't need the cloud to do that unless you're really, really huge, to the point that other costs are dominating anyhow.
> I wrote CloudFormation scripts that handle our basic use cases and use parameters.
scripts will run on things other than the cloud. The thing that made that difference? It may have been you, not the cloud.
> > Even compared to bare-metal, the difference isn't all it's chalked up to be - the actual rusty box doesn't need a lot of TLC, and you can hire companies for that too,*
> Can I hire a company to put in 20 additional servers every night when our processing queue increases and take them out when it decreases?
Nope; you can't - and that's a real upside to a cloud. I'm not opposed to using the cloud; merely saying that personnel costs savings are nonsense, however. In any case; you cannot easily do this with physical hosts. Then again; most people probably don't need to. If your workload is merely a little spiky (like day vs. night) then the extra costs due to the cloud will be greater even if you spin down instances sometimes; and if you do that, you're spending time to do so, which undercuts the clouds other selling point "I don't want to manager all that junk".
On HA: You're listing scaling, not HA. You don't need to do anything special to make interchangable, largely stateless machines "HA" no matter your infrastructure. HA is relevant if you have any single points of failure, and how you avoid the consequences of having those; e.g. having a live database replica or whatever. But sure: this 'll be easier on some clouds!
> > Almost all actual downtime is not because your physical infrastructure went down in ways that most HA could have prevented.
> HA can be as simple as a rogue query killed an app/web server (been there done that). I got an alert that the web server was unhealthy. By the time I woke up well, I got another alert saying that a new instance was brought up. I went back to sleep and investigated in the morning. Not to mention the automatic updates performed on RDS instances or the updates that you don’t have to care about with managed services.
So... why does any of that need the cloud? Sounds like a plain old process or container needed restarting. That's just basic server stuff servers have been doing for... well, longer than the cloud even existed.
But the point isn't that the cloud is a bad idea. It's that people ascribe frankly absurd benefits to it, when many of those are either unrelated to being in a cloud, and some are wildly overstated, specifically the management costs. Not that there aren't upsides: but if you're saving 170k on network engineering personnel costs; well, you didn't need the cloud to do that unless you're really, really huge, to the point that other costs are dominating anyhow.
scripts will run on things other than the cloud. The thing that made that difference? It may have been you, not the cloud.
Can you run a script that creates a VM that autoscales your build server from a size that runs 1 build to 50 builds simultaneously? That’s the equivalent of CodeBuild.
* On HA: You're listing scaling, not HA. You don't need to do anything special to make interchangable, largely stateless machines "HA" no matter your infrastructure. HA is relevant if you have any single points of failure, and how you avoid the consequences of having those; e.g. having a live database replica or whatever. But sure: this 'll be easier on some clouds!*
HA and scaling are conceptually different, but in the cloud they are practically the same thing.
An autoscaling group for instance that you set for a min/max of two across availability zones will ensure that you have two instances running whether your computer goes down or the entire zone goes down. It’s just a matter of rules what it scales based on.
The only practical difference between autoscaling and HA is scaling across AZ’s (overly simplified).
Then again; most people probably don't need to. If your workload is merely a little spiky (like day vs. night) then the extra costs due to the cloud will be greater even if you spin down instances sometimes; and if you do that, you're spending time to do so, which undercuts the clouds other selling point "I don't want to manager all that junk".
A “little” spiky going from 1 VM to 20? And that’s just one workload with Windows servers. We have other workloads where we need to reindex our entire database from Mysql to ElasticSearch and it automatically autoscales Mysql read replicas. How much would it costs to keep multiple read replicas up just to have throughout you only need once a month?
How much “management” do you think autoscaling based on an SQS is? You set up a rule that says when x number of messages are in the queue, scale up to the maximum, when their are less than y messages you scale down. This happens when everyone is sleep. Of course we do this with CloudFormation but even clicking in the console it literally takes minutes. It took me about 2 hours to create the CF template to do it and I was new at the time.
There are all sorts of spiky workloads. Colleges are well known for having spiky workloads during registration for instance.
but if you're saving 170k on network engineering personnel costs; well, you didn't need the cloud to do that unless you're really, really huge, to the point that other costs are dominating anyhow.
It doesn’t take being huge to get complicated:
- networking infrastructure
- permissions
- load balancers
- VMs (of course we need 20x the capacity just to handle peak)
- Mysql server + 1 read replica. Again we would need 3 or 4 more just to sit idle most of the time.
- enough servers to run our “serverless” workloads at peak.
- a server for our messaging system (instead of SNS/SQS)
- an SFTP server (instead of a managed solution)
- a file server with backups (instead of just using S3)
- whatever the open source equivalent of just being able to query and analyze data on S3 using Athena.
- a monitoring and alerting system (instead of CloudWatch)
- an ElasticSearch cluster
- Some type of OLAP database to take the place of Redshift.
- a build server instead of just using CodeBuild
- of course we can’t host our own CDN or just host a bunch of files in S3 and serve them up as a website and all of the server APIs hosted in lambda.
- We would have to host our own domain server.
- we would Still need load balancers.
And...Did I mention that most of this infrastructure would need to be duplicated for four different environments - DEV, QA, STG, and Prod?
And none of this is overly complicated with AWS, on prem we would have to have someone to manage it. Imagine managing all of that at a colo? We would definitely need someone on call. The only things that would go down and that we would have to do anything about are our web and API servers.
While there might be some thrashing with them being killed and brought back up until we figured out what was wrong, autoscaling would at least keep them up.
Can you run a script that creates a VM that autoscales your build server from a size that runs 1 build to 50 builds simultaneously? That’s the equivalent of CodeBuild.
* On HA: You're listing scaling, not HA. You don't need to do anything special to make interchangable, largely stateless machines "HA" no matter your infrastructure. HA is relevant if you have any single points of failure, and how you avoid the consequences of having those; e.g. having a live database replica or whatever. But sure: this 'll be easier on some clouds!*
HA and scaling are conceptually different, but in the cloud they are practically the same thing.
An autoscaling group for instance that you set for a min/max of two across availability zones will ensure that you have two instances running whether your computer goes down or the entire zone goes down. It’s just a matter of rules what it scales based on.
The only practical difference between autoscaling and HA is scaling across AZ’s (overly simplified).
Then again; most people probably don't need to. If your workload is merely a little spiky (like day vs. night) then the extra costs due to the cloud will be greater even if you spin down instances sometimes; and if you do that, you're spending time to do so, which undercuts the clouds other selling point "I don't want to manager all that junk".
A “little” spiky going from 1 VM to 20? And that’s just one workload with Windows servers. We have other workloads where we need to reindex our entire database from Mysql to ElasticSearch and it automatically autoscales Mysql read replicas. How much would it costs to keep multiple read replicas up just to have throughout you only need once a month?
How much “management” do you think autoscaling based on an SQS is? You set up a rule that says when x number of messages are in the queue, scale up to the maximum, when their are less than y messages you scale down. This happens when everyone is sleep. Of course we do this with CloudFormation but even clicking in the console it literally takes minutes. It took me about 2 hours to create the CF template to do it and I was new at the time.
There are all sorts of spiky workloads. Colleges are well known for having spiky workloads during registration for instance.
but if you're saving 170k on network engineering personnel costs; well, you didn't need the cloud to do that unless you're really, really huge, to the point that other costs are dominating anyhow.
It doesn’t take being huge to get complicated:
- networking infrastructure
- permissions
- load balancers
- VMs (of course we need 20x the capacity just to handle peak)
- Mysql server + 1 read replica. Again we would need 3 or 4 more just to sit idle most of the time.
- enough servers to run our “serverless” workloads at peak.
- a server for our messaging system (instead of SNS/SQS)
- an SFTP server (instead of a managed solution)
- a file server with backups (instead of just using S3)
- whatever the open source equivalent of just being able to query and analyze data on S3 using Athena.
- a monitoring and alerting system (instead of CloudWatch)
- an ElasticSearch cluster
- Some type of OLAP database to take the place of Redshift.
- a build server instead of just using CodeBuild
- of course we can’t host our own CDN or just host a bunch of files in S3 and serve them up as a website and all of the server APIs hosted in lambda.
- We would have to host our own domain server.
- we would Still need load balancers.
And...Did I mention that most of this infrastructure would need to be duplicated for four different environments - DEV, QA, STG, and Prod?
And none of this is overly complicated with AWS, on prem we would have to have someone to manage it. Imagine managing all of that at a colo? We would definitely need someone on call. The only things that would go down and that we would have to do anything about are our web and API servers.
While there might be some thrashing with them being killed and brought back up until we figured out what was wrong, autoscaling would at least keep them up.
> If you want to use managed services (Fargate, Aurora, DynamoDB, etc) at AWS, it's even more expensive!
More expensive than what? Running similar services yourself?
More expensive than what? Running similar services yourself?
Indeed, the link makes AWS look so affordable even ignoring all the value the author ignores (which is most of it) that it almost looks like part of a stealth AWS ad campaign.
Not to mention the automation and yes you can cut back staff. You can also outsource a lot of the menial networking support to an MSP.
The reason I use AWS is because I can be fairly certain, that their uptime is quite high (not 100%, but it's more than I can probably guarantee if I were to run on an own server), that their eco-system is huge and they connect to each other.
Being a "solo hacker" developing multiple websites, I need to prioritize where I put my time in. Thus my "default" choice is using Elastic Beanstalk to deploy websites, be it PHP, Python or something else. I can easily add environment variables, let it connect to RDS (database), scale it up (more servers or bigger servers), I can add SSL certificates, I can add a CDN (Cloudfront), I can add separate data storage (S3) all with a few clicks and from one (albeit super-huge and need-to-get-used-to) dashboard.
Alternatives: I don't trust Google services for anything critical (they tend to shut down easily or they might just change their pricing drastically - e.g. in Google Maps), Azure might be an alternative, just never had the time to check it out intensively.
For "fixed ceiling" projects, I use Digital Ocean (because unfortunately AWS mostly doesn't offer a budget ceiling, just alarms).
Being a "solo hacker" developing multiple websites, I need to prioritize where I put my time in. Thus my "default" choice is using Elastic Beanstalk to deploy websites, be it PHP, Python or something else. I can easily add environment variables, let it connect to RDS (database), scale it up (more servers or bigger servers), I can add SSL certificates, I can add a CDN (Cloudfront), I can add separate data storage (S3) all with a few clicks and from one (albeit super-huge and need-to-get-used-to) dashboard.
Alternatives: I don't trust Google services for anything critical (they tend to shut down easily or they might just change their pricing drastically - e.g. in Google Maps), Azure might be an alternative, just never had the time to check it out intensively.
For "fixed ceiling" projects, I use Digital Ocean (because unfortunately AWS mostly doesn't offer a budget ceiling, just alarms).
I've used various hosts over the last 15 years in the UK and I honestly can't remember any downtime from anyone else. I do remember lots of AWS + Azure downtime though. Remember Netflix going down?
I'm not really sure where you're coming from.
As someone who's been doing this for years, the old days when you had a £10 virtual machine that could handle tens of concurrent requests don't seem to be handled by the low price AWS/Azure servers. Let's not forget, tens of concurrent requests is 100,000s of monthly users, a major win for most startups.
And even the expensive cloud options are utterly rubbish compared to what you can get for a bare metal, I like to compare the two as you can get a Ferrari for £1000 a year, or you can get a bicycle for £1000 but don't worry, it's a "cloud" bicycle so definitely go with that.
I am constantly having problems with an Azure app at the moment timing out because of some poorly optimized queries that are being throttled because they somehow exceed the ever mysterious DTU. On my laptop, a 5 year old laptop, the queries take 10ms. So it's is impossible to run on Azure because "reasons" unless I pay a ridiculous amount of money to upgrade the SQL server. Yes, I could fix the query, but why should I be spending time on that? They're part of the admin system and are simple reports.
Another app I worked on we ended up having to pay £1000 a year just so we could have multiple-domains + SSL, when we expecting to pay £80 p/y.
Personally, every time I use cloud servers I regret it. I keep trying or clients really want it, and can understand the lower devops cost and that with bare metal, every 5-10 years I need to migrate the app to get a better server, but I honestly find the cloud servers so completely underpowered for the cost. I don't understand how they've got so popular. Basic devops for most startups is a day or two task once every few years. Write a deploy script, backup script once, and almost forget about it.
(I absolutely love s3 though for backups though)
I'm not really sure where you're coming from.
As someone who's been doing this for years, the old days when you had a £10 virtual machine that could handle tens of concurrent requests don't seem to be handled by the low price AWS/Azure servers. Let's not forget, tens of concurrent requests is 100,000s of monthly users, a major win for most startups.
And even the expensive cloud options are utterly rubbish compared to what you can get for a bare metal, I like to compare the two as you can get a Ferrari for £1000 a year, or you can get a bicycle for £1000 but don't worry, it's a "cloud" bicycle so definitely go with that.
I am constantly having problems with an Azure app at the moment timing out because of some poorly optimized queries that are being throttled because they somehow exceed the ever mysterious DTU. On my laptop, a 5 year old laptop, the queries take 10ms. So it's is impossible to run on Azure because "reasons" unless I pay a ridiculous amount of money to upgrade the SQL server. Yes, I could fix the query, but why should I be spending time on that? They're part of the admin system and are simple reports.
Another app I worked on we ended up having to pay £1000 a year just so we could have multiple-domains + SSL, when we expecting to pay £80 p/y.
Personally, every time I use cloud servers I regret it. I keep trying or clients really want it, and can understand the lower devops cost and that with bare metal, every 5-10 years I need to migrate the app to get a better server, but I honestly find the cloud servers so completely underpowered for the cost. I don't understand how they've got so popular. Basic devops for most startups is a day or two task once every few years. Write a deploy script, backup script once, and almost forget about it.
(I absolutely love s3 though for backups though)
If you can run your app on a £10 VPS then you are not the target market for the big clouds. That's why it doesn't make sense for you. Larger users have very different priorities.
This really isn't true. People use them for everything. Tiny little enterprise apps with 10s of users. Startups with mere 1,000s of uniques per month. Side projects. Everything.
If you look at Microsoft's strategy with C#/Azure you'll see that every single tutorial uses deploying on Azure. Their instructions for how to deploy to a real server in their tutorial is pretty much "we're not telling you, figure it out yourself, sucker".
Their new business strategy is to get the entire of their userbase paying massively over the odds for hosting small apps.
And, in reality, almost every app by any startup you could go and look at on producthunt (and probably most of ycombinator startups) have such low volumes of usage they could run on a $5 VPS. But I bet all of them are on AWS (there's even a few comments on this thread that basically say "start on AWS, then move").
Everyone is using cloud now, go to any meetup and ask. Or simply look at the sales figures.
If you look at Microsoft's strategy with C#/Azure you'll see that every single tutorial uses deploying on Azure. Their instructions for how to deploy to a real server in their tutorial is pretty much "we're not telling you, figure it out yourself, sucker".
Their new business strategy is to get the entire of their userbase paying massively over the odds for hosting small apps.
And, in reality, almost every app by any startup you could go and look at on producthunt (and probably most of ycombinator startups) have such low volumes of usage they could run on a $5 VPS. But I bet all of them are on AWS (there's even a few comments on this thread that basically say "start on AWS, then move").
Everyone is using cloud now, go to any meetup and ask. Or simply look at the sales figures.
The thing is, maybe you're overpaying spending $100/month instead of $10/month, but those are both small numbers. Simply not having to standup a DB/server/getting certs/debug if something goes wrong somewhere/etc is worth more in my time than a few $100. If you're very small scale you can also use the "always free" tier of VMs.
I really like that for my side projects/solo hacks I don't have to think about anything other "here is my code that does the thing that I think is cool" and all the deployment/serving/data layer/ml model inference/etc just kind of happens.
For small/very early startups it's also similar. At least in the US, it's not the extra $5k/year you spend on cloud hosting vs doing it yourself that kills you; for many engineers that less than a week of comp.
At the same time, when you are large and spending $100s of k on your hosting then sure, look into alternatives and migrate if needed. This is where "start on AWS, then move" comes from.
I really like that for my side projects/solo hacks I don't have to think about anything other "here is my code that does the thing that I think is cool" and all the deployment/serving/data layer/ml model inference/etc just kind of happens.
For small/very early startups it's also similar. At least in the US, it's not the extra $5k/year you spend on cloud hosting vs doing it yourself that kills you; for many engineers that less than a week of comp.
At the same time, when you are large and spending $100s of k on your hosting then sure, look into alternatives and migrate if needed. This is where "start on AWS, then move" comes from.
I guess the point I was trying to make is that all that "extra" stuff isn't really much extra work at all and the same magic deploy tools could be just as easy on a VPS/bare-metal, but aren't because of $$profit$$ making it better that they don't make it super-easy to deploy on VPS/bare-metal.
I can make a script to do it all for me (and have in the past), but it's a bit of work.
That's really why the title of the article is apt being "the Amazon Premium", and not "the Amazon Rip-off".
I can make a script to do it all for me (and have in the past), but it's a bit of work.
That's really why the title of the article is apt being "the Amazon Premium", and not "the Amazon Rip-off".
Honestly if you are deploying an application to production, hopefully you have a level of knowledge beyond "click here, then click here'.
The other side of this is that documentation is hard, and Microsoft documentation is frequently lacking or out of date, especially for anything off the beaten path. I use IoT Hub for messaging, and it's documentation is scant, many limitations and failure modes are found by trial and error.
The other side of this is that documentation is hard, and Microsoft documentation is frequently lacking or out of date, especially for anything off the beaten path. I use IoT Hub for messaging, and it's documentation is scant, many limitations and failure modes are found by trial and error.
I don't understand where you would get this info from then? Tutorials provided by the framework makers themselves are pretty standard.
Magically absorb the knowledge in an incantation to the Redmond Gods in the ether?
Magically absorb the knowledge in an incantation to the Redmond Gods in the ether?
haha, sometimes it feels that way, for their less-popular commercial services documentation is lacking and you can't even read the source!
On reflection my comment about level of skill is unfair, what I should have said is: generally the information is there, but it's not always in the form of follow-able 'How-to' guides, often it's a dry detailed documentation. If you need something other than the standard setup, it is up to you to know what you are looking for and put 2 and 2 together. Of course it is by no means easy.
They do have reasonable guidance on how to deploy ASP.net core on Linux, but it can be lacking for lesser used stuff
On reflection my comment about level of skill is unfair, what I should have said is: generally the information is there, but it's not always in the form of follow-able 'How-to' guides, often it's a dry detailed documentation. If you need something other than the standard setup, it is up to you to know what you are looking for and put 2 and 2 together. Of course it is by no means easy.
They do have reasonable guidance on how to deploy ASP.net core on Linux, but it can be lacking for lesser used stuff
If you’re that small, you can probably stay under the always free lambda tier.
Honestly, why chime in with that? Not only is it sort of sneering (if you're that small), it's wrong and counter to the most practical advice we give out here, use what you know. It's hard enough starting a business without learning a whole new tech stack.
Ironically, we were having the discussion here yesterday about a guy with a failed startup saying he'd wasted the opportunity because he used too much new stuff and spent all his time on that instead of iterating the actual product. Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.
And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality. Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.
Ironically, we were having the discussion here yesterday about a guy with a failed startup saying he'd wasted the opportunity because he used too much new stuff and spent all his time on that instead of iterating the actual product. Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.
And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality. Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.
Not only is it sort of sneering (if you're that small)
Lambdas always free tier is 1 million requests per month and 400Gb/seconds per month.
https://aws.amazon.com/lambda/pricing/
It takes a lot to have any meaningful lambda bill. That in no way is meant to be an insult. If you’re spending even $100/month on lambda, you’re running a successful website.
It's hard enough starting a business without learning a whole new tech stack.
If you know how to write a program and create a function that takes two parameters - a payload and context argument - you know lambda. You can write it in the console if you’re using Python or Node and just run it from there.
Even if you don’t want to learn that, you can host your standard C#/WebAPI, Python/Django/Flask, Node/Express API on lambda just by adding code and changing your endpoint.
Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.
There is also code to wrap a Rails app and run it on lambda....
And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality
Yes. Especially in today’s world of client side frameworks and microservices, you host your static assets like Javascript, css, and HTML in S3 and call server side APIs hosted on lambda.
Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.
Well, two things. What’s more likely to die in two years / your startup or AWS?
Second, you can host your standard API just by adding four lines of code and you don’t have to make any changes to host it in Docker or on a VM.
There is a way to do this in every language that lambda natively supports. For instance for Node:
https://github.com/awslabs/aws-serverless-express
I have a Node/Express app that gets deployed to both lambda and Fargate (Serverless Docker) when I push it. The only difference is that lambda uses one entry point and Docker uses another.
I’ve done something similar with C#
https://aws.amazon.com/blogs/developer/deploy-an-existing-as...
Where it is deployed to both lambda and a Windows server running IIS.
I can be a lot more productive with all the resources that AWS has to offer when all I have to do is provision resources with a CloudFormation template and instead of doing the grunt work of infrastructure administration I can concentrate on writing software.
Lambdas always free tier is 1 million requests per month and 400Gb/seconds per month.
https://aws.amazon.com/lambda/pricing/
It takes a lot to have any meaningful lambda bill. That in no way is meant to be an insult. If you’re spending even $100/month on lambda, you’re running a successful website.
It's hard enough starting a business without learning a whole new tech stack.
If you know how to write a program and create a function that takes two parameters - a payload and context argument - you know lambda. You can write it in the console if you’re using Python or Node and just run it from there.
Even if you don’t want to learn that, you can host your standard C#/WebAPI, Python/Django/Flask, Node/Express API on lambda just by adding code and changing your endpoint.
Several comments focused on it'd just be better to start off with a Rails app, it's quick, it's easy and it's understood.
There is also code to wrap a Rails app and run it on lambda....
And really, is lambda appropriate for anything more than a very focused niche app? I mean, try writing all the usual things you need to slot into a SaaS on it, billing, user management, signup, admin reports, admin functionality, emails, multi-tenancy, custom domains, etc. etc. Even a 20 user, internal, enterprise app needs most of that functionality
Yes. Especially in today’s world of client side frameworks and microservices, you host your static assets like Javascript, css, and HTML in S3 and call server side APIs hosted on lambda.
Put on top of that, you're forced to use a particular, proprietary tech stack and lock yourself into something that might very well die in a couple of years.
Well, two things. What’s more likely to die in two years / your startup or AWS?
Second, you can host your standard API just by adding four lines of code and you don’t have to make any changes to host it in Docker or on a VM.
There is a way to do this in every language that lambda natively supports. For instance for Node:
https://github.com/awslabs/aws-serverless-express
I have a Node/Express app that gets deployed to both lambda and Fargate (Serverless Docker) when I push it. The only difference is that lambda uses one entry point and Docker uses another.
I’ve done something similar with C#
https://aws.amazon.com/blogs/developer/deploy-an-existing-as...
Where it is deployed to both lambda and a Windows server running IIS.
I can be a lot more productive with all the resources that AWS has to offer when all I have to do is provision resources with a CloudFormation template and instead of doing the grunt work of infrastructure administration I can concentrate on writing software.
[deleted]
I have recently started a project of deploying kubernetes cluster at home and hosting some personal projects there. My top findings are:
1) bare metal is dramatically faster than most cloud instances. Many of us might have collection of old hardware that, by performance, would be equivalent to >$1,000 in monthly cloud fees.
2) If from the start of your career you have only dealt with modern cloud/Azure/AWS, there is a whole body of knowledge you are missing. Acquiring it from scratch is time consuming, and you are never quite sure you are doing the right thing. As a developer, I don't pay the AWS bill, so it would be more rational to spend my time on learning ML or something else that will be reflected in my paycheck.
1) bare metal is dramatically faster than most cloud instances. Many of us might have collection of old hardware that, by performance, would be equivalent to >$1,000 in monthly cloud fees.
2) If from the start of your career you have only dealt with modern cloud/Azure/AWS, there is a whole body of knowledge you are missing. Acquiring it from scratch is time consuming, and you are never quite sure you are doing the right thing. As a developer, I don't pay the AWS bill, so it would be more rational to spend my time on learning ML or something else that will be reflected in my paycheck.
So how do you do routing? I have an old Linux machine that I haven’t been using and don’t know how I can securely and efficient route that into the internet (don’t want to use my ip for connecting to Redis for example) which is why I as thinking of using some vpn hosted on gcp.
Any advice would be greatly appreciated. Can’t handle the monthly payments to gcp only to get a raspberry pi level of compute.
Any advice would be greatly appreciated. Can’t handle the monthly payments to gcp only to get a raspberry pi level of compute.
I host a Plex instance on a server on my home network, and struggled with setting up external access because my ISP at the time had a double NAT. What I ended up doing was renting a super cheap VPS from TINYKVM (https://tinykvm.com/) which I pay $15 a year for.
I point my DNS at the VPS, then I use the Zerotier VPN to tunnel traffic to my home server (Zerotier is embarrassingly easy to set up - I struggled for a while with OpenVPN but gave up). I use iptables packet forwarding to route traffic on port 443 or 80 from the VPS over the VPN to my local server, which is running an Nginx instance to serve the traffic.
This works perfectly for me 99% of the time. Main pain points have been the unreliability of hosting a server locally where Internet can die and power drop off, and a small amount of sysadmining on the VPS, which is so tiny that I have to be mindful of disk space (e.g. removing old Linux upgrade images every so often, etc). All told though, it works surprisingly seamlessly, and there's a certain smugness to knowing traffic from my smart TV goes all the way round the world and back through a VPN just to talk to my server sitting one foot below the TV.
I point my DNS at the VPS, then I use the Zerotier VPN to tunnel traffic to my home server (Zerotier is embarrassingly easy to set up - I struggled for a while with OpenVPN but gave up). I use iptables packet forwarding to route traffic on port 443 or 80 from the VPS over the VPN to my local server, which is running an Nginx instance to serve the traffic.
This works perfectly for me 99% of the time. Main pain points have been the unreliability of hosting a server locally where Internet can die and power drop off, and a small amount of sysadmining on the VPS, which is so tiny that I have to be mindful of disk space (e.g. removing old Linux upgrade images every so often, etc). All told though, it works surprisingly seamlessly, and there's a certain smugness to knowing traffic from my smart TV goes all the way round the world and back through a VPN just to talk to my server sitting one foot below the TV.
Interesting setup. Have you heard of Cloudflare Argo by chance? Was thinking of using that, although this seems like a perfectly fine setup itself.
I am not entirely clear on your question, but I am avoiding reliance on any third-party service to keep it a clean experiment.
I have it easy for external access -> static IP and a Gigabit connection without NAT, so all I have to do is setup ports on my router.
For my services like databases and Redit, I expose them only on Cluster-IP, and them you use kubeDNS and get something like 'redis.default.svc.cluster.local'.
If I need to access them for development purposes, I use kubectl-proxy, but a VPN is also an option.
I have it easy for external access -> static IP and a Gigabit connection without NAT, so all I have to do is setup ports on my router.
For my services like databases and Redit, I expose them only on Cluster-IP, and them you use kubeDNS and get something like 'redis.default.svc.cluster.local'.
If I need to access them for development purposes, I use kubectl-proxy, but a VPN is also an option.
If your DNS service has an API, you can just have your home server update it’s own record when it detects a change. It may be down for a little while while DNS propagates though.
Edit: I think I misunderstood, but will leave this up anyway.
Edit: I think I misunderstood, but will leave this up anyway.
> started a project of deploying kubernetes cluster at home and hosting
I am also thinking along these lines but not sure about reasonable paths to try. Care to share some pointers on your setup? Basically looking to serve web pages from a server cluster at home.
I am also thinking along these lines but not sure about reasonable paths to try. Care to share some pointers on your setup? Basically looking to serve web pages from a server cluster at home.
> Alternatives: I don't trust Google services for anything critical
In the case of GCP, the product is the product rather than the user, so there's a greater expectation that it'll not get shut down or that pricing will change easily.
Further, they're obviously committed to being competitive with AWS which requires that they behave similarly.
In the case of GCP, the product is the product rather than the user, so there's a greater expectation that it'll not get shut down or that pricing will change easily.
Further, they're obviously committed to being competitive with AWS which requires that they behave similarly.
On the other hand.... see maps.
On the other hand of what?
Maps is a consumer facing service for which revenue is derived from the data collected and advertising. That business model is nothing like selling commodity compute services to enterprises.
If anything, I think GSuite is a portent of how Google will treat GCP. That is, the set of products has been relatively and consistently stable for quite some time, which has driven fairly decent adoption into the enterprise market.
There've a couple of product shifts, specifically around voice, hangout, and chat, but that seems to settled at this point.
I speculate that the changes that have raised the most ire have been around gmail, which I suspect has a fragmented product management team as it's both a free consumer service getting monetized through ad revenue as well as an enterprise product sold directly to customers.
Maps is a consumer facing service for which revenue is derived from the data collected and advertising. That business model is nothing like selling commodity compute services to enterprises.
If anything, I think GSuite is a portent of how Google will treat GCP. That is, the set of products has been relatively and consistently stable for quite some time, which has driven fairly decent adoption into the enterprise market.
There've a couple of product shifts, specifically around voice, hangout, and chat, but that seems to settled at this point.
I speculate that the changes that have raised the most ire have been around gmail, which I suspect has a fragmented product management team as it's both a free consumer service getting monetized through ad revenue as well as an enterprise product sold directly to customers.
Maps is a consumer facing service for which revenue is derived from the data collected and advertising. That business model is nothing like selling commodity compute services to enterprises.
So selling access to an API that businesses depend on is nothing like selling access to other resources that businesses depend on?
https://gadgets.ndtv.com/apps/features/google-maps-apis-new-...
Prices have gone up as much as 14 times in certain scenarios
So selling access to an API that businesses depend on is nothing like selling access to other resources that businesses depend on?
https://gadgets.ndtv.com/apps/features/google-maps-apis-new-...
Prices have gone up as much as 14 times in certain scenarios
That's correct.
Selling access to the Google maps API is wholly superfluous to Google's business model. Certainly, it doesn't clock anywhere near a billion in revenue.
Prices have gone up because Google doesn't have any interest in sustaining the product. The total addressable market is miniscule by comparison to cloud or even enterprise productivity apps.
Selling access to the Google maps API is wholly superfluous to Google's business model. Certainly, it doesn't clock anywhere near a billion in revenue.
Prices have gone up because Google doesn't have any interest in sustaining the product. The total addressable market is miniscule by comparison to cloud or even enterprise productivity apps.
Shouldn’t that also give you pause that Google is known not to care about products that are not its main revenue driver and will abandon them?
It doesn’t matter how large the “addressable market” is if Google doesn’t have the will or talent to compete. Social is also a large market, how did that work out for Google?
Isn’t that the entire pitch of every startup? “If we just get 10% of the market, we will be huge”.
And “revenue” is meaningless is GCP profitable? There was a submission on the front page of HN yesterday about Google’s management not being happy about how GCP waS doing and they have three years to turn it around or “lose funding”.
It doesn’t matter how large the “addressable market” is if Google doesn’t have the will or talent to compete. Social is also a large market, how did that work out for Google?
Isn’t that the entire pitch of every startup? “If we just get 10% of the market, we will be huge”.
And “revenue” is meaningless is GCP profitable? There was a submission on the front page of HN yesterday about Google’s management not being happy about how GCP waS doing and they have three years to turn it around or “lose funding”.
The lack of ease with which one gets into and out of the cloud computing business will preclude Google from killing off the GCP products with the same wanton disregard for customers as they demonstrated with social and others.
At worst, they'll sell off or divest the business entirely. They'll expose themselves to tremendous liability if they attempt to wind it down like their other properties.
At worst, they'll sell off or divest the business entirely. They'll expose themselves to tremendous liability if they attempt to wind it down like their other properties.
It doesn't even matter as much if Google gets out of "cloud" as it matters if Google decides to kill one of its cloud offering that you depend on. If I were on GCP and building a service that is using 5 Google features one of them happened to be the Maps API and the other 4 came under the "Cloud" umbrella, it would be little solace that Maps wasn't part of GCP when they raised the prices 15x.
On the other hand, AWS announced something as minor as that they were discontinuing one form of accessing publicly available S3 urls that was deprecated in 2010 for the newer one. After a vocal outcry from the internet about how many websites that it would break, they backtracked:
https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...
On the otherhand, how many Urls were broken when Google discontinued it's link shorterning service?
Google has shown time and time again that it can't be relied upon for any important infrastructure.
On the other hand, AWS announced something as minor as that they were discontinuing one form of accessing publicly available S3 urls that was deprecated in 2010 for the newer one. After a vocal outcry from the internet about how many websites that it would break, they backtracked:
https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...
On the otherhand, how many Urls were broken when Google discontinued it's link shorterning service?
Google has shown time and time again that it can't be relied upon for any important infrastructure.
I get your point, but you keep supporting it with examples of products that are well outside the auspices of the GCP business unit.
Yes, it's true that Google capriciously kills products for various business reasons without any regard for customer sentiment. My counter-point to that is that GCP products exist under different conditions, specifically legal and business realities. They have contractual obligations to corporations with very large legal departments. There is also the business reality that if they're going to compete with AWS and Azure then to be competitive they have to behave in similar ways. If they piss on customer sentiment as they have with their b2c services, they'll fail and I think this is very clear to them.
That's not to say that they aren't already failing. As I mentioned up thread, their support is laughably bad and when pressed about that fact, they don't seem to actually acknowledge how that could be a problem.
Yes, it's true that Google capriciously kills products for various business reasons without any regard for customer sentiment. My counter-point to that is that GCP products exist under different conditions, specifically legal and business realities. They have contractual obligations to corporations with very large legal departments. There is also the business reality that if they're going to compete with AWS and Azure then to be competitive they have to behave in similar ways. If they piss on customer sentiment as they have with their b2c services, they'll fail and I think this is very clear to them.
That's not to say that they aren't already failing. As I mentioned up thread, their support is laughably bad and when pressed about that fact, they don't seem to actually acknowledge how that could be a problem.
(Article author here) Unfortunately my experiences with GCP have been fairly mixed. One place I worked had a number of problems with managed databases going AWOL on GCP.
Also the GCP panel is truly awful, much much worse than the AWS console.
Also the GCP panel is truly awful, much much worse than the AWS console.
My experiences with GCP have been pretty good, most of those experiences were with GKE. In many ways, their products are superior to AWS. At the very least, they have a different approach to offering cloud services rather than simply attempting to copy AWS offerings.
However, their support is absolute shit and they're absolutely devoid of the customer first mentality that AWS demonstrates.
However, their support is absolute shit and they're absolutely devoid of the customer first mentality that AWS demonstrates.
I have to wonder if this depends on one's familiarity with the respective products. I started with GCP and then used AWS and found the latter to be way more confusing and convoluted than GCP. The forced daily logouts on AWS added another element of hassle to the mix.
https://www.crn.com/news/cloud/google-reportedly-set-ambitio...
Doesn't matter what is the product. They're ready to kick GCP to the curb if it doesn't perform up to expectations. I wouldn't host anything important on it.
Doesn't matter what is the product. They're ready to kick GCP to the curb if it doesn't perform up to expectations. I wouldn't host anything important on it.
I mostly work in AWS. I was recently creating an environment and a deployment mechanism for a fairly simple app that was initially developed in Azure. The client was surprised because the cost estimates previous devs gave him (based on Azure) were a lot lower. I don't know what their Azure architecture was - I had to come up with AWS stuff from scratch. Maybe they omitted certain things like backup, DR, auto scaling etc from their initial estimate or maybe it is worth looking at Azure. It is not the first time I'm hearing something along the lines of "Azure is cheaper".
I frequently see folks assert cloud vendor X is cheaper and present data comparing long-term reserved instances to on-demand.
How does your development cycle look like? For example, you decide you want to add S3 or Redis. So you add it to some yaml, or do it in the amazon console. How do you develop/test/debug the code that would use these services, locally, before pushing that to production?
I try to abstract something like the filesystem. So I would use something like Flysystem for PHP, or pyfilesystem for Python where an environment variable would decide whether it's saved to local disk or S3.
For some more difficult use cases I have to be pragmatic and decide on the circumstances. E.g. when needing presigned URLs (which is an S3 feature) I will use the native S3 SDK, it's not the cleanest way, but it gets the job done. Only when it starts getting annoying / slowing the development speed will I abstract it away. Security is important and I keep a high standard there, but when it comes to clean abstraction I will occasionally make compromises, knowing that often we will only need one implementation or that feature might be thrown away (I work freelance mostly with startups or SMEs).
For some more difficult use cases I have to be pragmatic and decide on the circumstances. E.g. when needing presigned URLs (which is an S3 feature) I will use the native S3 SDK, it's not the cleanest way, but it gets the job done. Only when it starts getting annoying / slowing the development speed will I abstract it away. Security is important and I keep a high standard there, but when it comes to clean abstraction I will occasionally make compromises, knowing that often we will only need one implementation or that feature might be thrown away (I work freelance mostly with startups or SMEs).
Thank you. BTW I agree with starting simple and only abstracting if necessary.
Amazon recommends "testing at scale", basically spin up a non-production version of your architecture and try it out.
Checkout Lightsail, been using it past 6 months. Power and freedom of digital ocean in AWS ecosystem.
To be clear, Lightsail is only "sorta" in the AWS ecosystem.
It is in the ecosystem in the sense that it ultimate boots up an EC2 instance. It is essentially a simplified skin on top of EC2 where AWS makes most of the decisions for you in regards to server setup, with simplified pricing. And it shows up as AWS on your bill, and can be billed along side other AWS services you use.
However, it isn't really in the ecosystem as I would argue. You can't connect it to networks and subnets you create. You don't have control over availability zones, so essentially everything (like talking to S3) is considered egress traffic. It is like a sandboxed AWS service, that sits out there on its own. It really isn't any different than using Linode or Digitial Ocean for your compute resource and using AWS for block storage, queuing, DNS, etc.. Other AWS services can't "see" your Lightsail instance, like they can see your EC2 instances. You can't connect EBS volumes to it, you can't take advantage of elastic IPs, or anything of that sort. It really is a separate service that sits on its' own, under the comfort of the AWS name.
If I remember right, even tags don't carry over between Lightsail and the rest of your AWS account. Again, Lightsail is fine as a service, but it is only "sort of" in the AWS ecosystem. It mostly sits on its own as an isolated service, backed by AWS.
It is in the ecosystem in the sense that it ultimate boots up an EC2 instance. It is essentially a simplified skin on top of EC2 where AWS makes most of the decisions for you in regards to server setup, with simplified pricing. And it shows up as AWS on your bill, and can be billed along side other AWS services you use.
However, it isn't really in the ecosystem as I would argue. You can't connect it to networks and subnets you create. You don't have control over availability zones, so essentially everything (like talking to S3) is considered egress traffic. It is like a sandboxed AWS service, that sits out there on its own. It really isn't any different than using Linode or Digitial Ocean for your compute resource and using AWS for block storage, queuing, DNS, etc.. Other AWS services can't "see" your Lightsail instance, like they can see your EC2 instances. You can't connect EBS volumes to it, you can't take advantage of elastic IPs, or anything of that sort. It really is a separate service that sits on its' own, under the comfort of the AWS name.
If I remember right, even tags don't carry over between Lightsail and the rest of your AWS account. Again, Lightsail is fine as a service, but it is only "sort of" in the AWS ecosystem. It mostly sits on its own as an isolated service, backed by AWS.
Thanks! I had taken a look when it first launched, but haven't looked at it since. Will try it out for the next (low-risk) project.
And none of the cost control.
Looks like you get charged if you go over the transfer allowance. Is there a way to automatically shut everything down to prevent that from happening? I like to go months without thinking about my server, knowing there's no way for a bot army to come around and ramp up my charges. I realize that's paranoid, but Amazon's competitors give me that option.
The other cool thing about Lightsail is that you can upgrade to EC2 should you need to go to the next level. I assume that the Lightsail instances are really some sort of EC2 instance that has a user friendly wrapper around it.
I suspect some of the premium is because "nobody ever got fired for buying AWS"
If you chose AWS and there's a security incident or hours-long outage, well, some of the best in the industry experienced the same thing, what can you do? Whereas if you chose Linode and they had exactly the same incident justly or otherwise, people might question your choice of a cut price $10/month provider.
If you chose AWS and there's a security incident or hours-long outage, well, some of the best in the industry experienced the same thing, what can you do? Whereas if you chose Linode and they had exactly the same incident justly or otherwise, people might question your choice of a cut price $10/month provider.
This is why a company i worked at about 3 years ago switched to AWS. I suggested using Digital Ocean and argued that we didn't need the complexity of AWS. But the CEO (who is so tech-illiterate that he struggles to send basic email, I am not even joking) has heard that everyone else uses AWS, so he wanted to use it too. Nobody was going to stop this guy from switching over to AWS. He wanted his company to rank among the cool kids that was built on AWS. He had heard of AWS and he trusts Amazon. He has never heard of Digital Ocean. His un-warranted opinion on switching to AWS was not worth fighting, so I let him switch as he desired. If any problem had ever come up with Digital Ocean, he would have pointed fingers at me and said "this wouldn't have happened with AWS, because they are AMAZON!".
And thus is why one small company dumps $1,000 a month into AWS when it could dump about $250 a month into Digital Ocean for comparable results. I know these are small numbers for many people on here, but it is a good illustration.
For what its worth. I was able to justify getting an AWS Certification because of this, which has been surprisingly helpful to have on my resume, even with companies that don't use AWS. It seems to be a fairly respected certification that demonstrates knowledge of cloud computing in general.
And thus is why one small company dumps $1,000 a month into AWS when it could dump about $250 a month into Digital Ocean for comparable results. I know these are small numbers for many people on here, but it is a good illustration.
For what its worth. I was able to justify getting an AWS Certification because of this, which has been surprisingly helpful to have on my resume, even with companies that don't use AWS. It seems to be a fairly respected certification that demonstrates knowledge of cloud computing in general.
Risk mitigation. Your CEO is probably juggling several issues from budget, hiring, sales, real estate etc. Some of these are probably high risk decisions that could sink the company. So the decision to go with AWS and not care about marginal cost savings are about eliminating a potential source of risk.
Dude, even for a small company the difference between $1000/mo and $250/mo is peanuts.
Maybe. But the decision heuristic here, which is "we pay 4x more for things because the CEO doesn't trust his people and like to micromanage things" can easily be fatal.
The company paid $1 instead of a quarter. Why the original poster even put up a fight about saving 750 dollars.... can't tell you. They probably spent more on keeping the kitchen stocked with coffee and soda each month.
The company is paying an extra $9000/year. With a boss like this, it's perfectly possible that they don't stock the kitchen with coffee and soda, because the CEO is already having somebody bring him his caramel macchiatos, so he doesn't care, and money's tight, because his people just do what the boss says instead of him trusting them to do the right thing.
Definitely the case at my company. A while ago we did an evaluation of different cloud providers and hosting solutions but it was pretty clear that “AWS” was the only acceptable answer. Which is probably ok. If you don’t have much experience in that space AWS is probably the reasonable and safe way to go.
As someone who's worked at many startups and look at this from that point of view.
Here are my key points:
- What makes aws great is NOT ec2... it's the ALB, route53, ecs/eks/fargate, redshift, rds, lambda, SQS, Kinesis, cloudwatch, cloudfront... ect. The last startup I was at our production MySql got corrupted... glad I paid for RDS and had their 5 minute interval backups to s3 that required 1 click.
- Nearly everyone is on aws, if aws is down the internet is down. The down time from aws is often hand waved away.
- With aws I have access to 3rd party services like snowflake and databricks. Hiring and training talent is more expensive then ease of use services.
- The cost to performance ratio had always been worth it for every startup* I worked at (double given aws gives baiscly a free year when you raise a round of vc funding). Its costly to dedicate staff/time to infrastructure. I had a real time data pipeline up in under 2 hrs using kensis and imply. Just having to set up somthing like kafka alone will take a multiplitude longer. *The exception being the startup that did billions of requests a day... I doubt they regretting paying aws to get to that point after 6ish years.
Here are my key points:
- What makes aws great is NOT ec2... it's the ALB, route53, ecs/eks/fargate, redshift, rds, lambda, SQS, Kinesis, cloudwatch, cloudfront... ect. The last startup I was at our production MySql got corrupted... glad I paid for RDS and had their 5 minute interval backups to s3 that required 1 click.
- Nearly everyone is on aws, if aws is down the internet is down. The down time from aws is often hand waved away.
- With aws I have access to 3rd party services like snowflake and databricks. Hiring and training talent is more expensive then ease of use services.
- The cost to performance ratio had always been worth it for every startup* I worked at (double given aws gives baiscly a free year when you raise a round of vc funding). Its costly to dedicate staff/time to infrastructure. I had a real time data pipeline up in under 2 hrs using kensis and imply. Just having to set up somthing like kafka alone will take a multiplitude longer. *The exception being the startup that did billions of requests a day... I doubt they regretting paying aws to get to that point after 6ish years.
> Nearly everyone is on aws, if aws is down the internet is down. The down time from aws is often hand waved away.
This is not even close to true. The 3rd answer has some numbers for you: https://www.quora.com/How-essential-is-AWS-to-the-internet
You’ll notice that the most dramatic responses come from other AWS devs like yourself (the first answer in the quora link). This is because as an AWS devotee you end up in an echo chamber.
This is not even close to true. The 3rd answer has some numbers for you: https://www.quora.com/How-essential-is-AWS-to-the-internet
You’ll notice that the most dramatic responses come from other AWS devs like yourself (the first answer in the quora link). This is because as an AWS devotee you end up in an echo chamber.
In my experience its true. When S3 went down my companies systems started malfunctioning, as well as many other vendors/system all over the internet including things like slack. Our customers were experiencing pain from multiple vendor failures. When our customers cant order lunch, run trello, or sent chat messages on slack they blame "the internet".
Note:I take offence to being called an AWS devotee, I have been in this space professionally for going on 13 years with nearly all of it in the startup space. The early years had to rely on collocation. To see startups struggle from things like hardware failure and delaying sales because they need more hardware is something I do not fondly remember.
Note:I take offence to being called an AWS devotee, I have been in this space professionally for going on 13 years with nearly all of it in the startup space. The early years had to rely on collocation. To see startups struggle from things like hardware failure and delaying sales because they need more hardware is something I do not fondly remember.
Apparently “the internet” to you is just hip startups if your name call outs are things like slack, trello, and lunch ordering. The startup echo chamber and the AWS echo chamber have a lot of overlap due to the elasticity you highlighted. That still doesn’t mean it’s representative of internet services.
When an AWS outage stops wire transfers, airline reservations, robot surgeons, etc, then it really will be “everyone”. Until then, try to give yourself some perspective so you can recognize when a product is mature enough that it’s time to start moving away from AWS.
> I have been in this space professionally for going on 13 years with nearly all of it in the startup space
This is why the siren call of instant infrastructure is so alluring to you. While your depth of knowledge for startup infra requirements is there, it does not transfer to large enterprises/campuses/etc where demand is very predictable and involving another company (Amazon) in your operations is nothing more than a liability.
Before you bemoan on-prem competing with Amazon’s world-class engineering org, consider that their priorities are spread across thousands of products all interacting with millions of customers. An outage that could destroy your business means nothing to them other than some reduced KPIs that month. While an on-prem solution might be worse in throughput, global latency, etc, the fate is in control of the org that can make sacrifices Amazon can’t (e.g. planned nightly outages).
When an AWS outage stops wire transfers, airline reservations, robot surgeons, etc, then it really will be “everyone”. Until then, try to give yourself some perspective so you can recognize when a product is mature enough that it’s time to start moving away from AWS.
> I have been in this space professionally for going on 13 years with nearly all of it in the startup space
This is why the siren call of instant infrastructure is so alluring to you. While your depth of knowledge for startup infra requirements is there, it does not transfer to large enterprises/campuses/etc where demand is very predictable and involving another company (Amazon) in your operations is nothing more than a liability.
Before you bemoan on-prem competing with Amazon’s world-class engineering org, consider that their priorities are spread across thousands of products all interacting with millions of customers. An outage that could destroy your business means nothing to them other than some reduced KPIs that month. While an on-prem solution might be worse in throughput, global latency, etc, the fate is in control of the org that can make sacrifices Amazon can’t (e.g. planned nightly outages).
* This is why the siren call of instant infrastructure is so alluring to you. While your depth of knowledge for startup infra requirements is there, it does not transfer to large enterprises/campuses/etc where demand is very predictable and involving another company (Amazon) in your operations is nothing more than a liability.*
Every large company has dozens of vendors that they “involve their operations”.
Every large company has dozens of vendors that they “involve their operations”.
The best part about companies that use the cloud is they don't have to deal with joyful sysadmin gatekeepers like you. That is like 75% of the reason most people use the cloud. To avoid dealing with gatekeepers such as you who "know better". The ones that slow every god damn project in the company down...
> This is why the siren call of instant infrastructure is so alluring to you. While your depth of knowledge for startup infra requirements is there, it does not transfer to large enterprises/campuses/etc where demand is very predictable and involving another company (Amazon) in your operations is nothing more than a liability.
My whole original post was from the startup point of view and I made that quite clear. I am more then happy to admit the enterprise space (from an infrastructure POV) is not my expertise. Your taking my points out of context. Even in my own example I added an astrisk of a startup who started moved to hybrid on-prem after six years.
My whole original post was from the startup point of view and I made that quite clear. I am more then happy to admit the enterprise space (from an infrastructure POV) is not my expertise. Your taking my points out of context. Even in my own example I added an astrisk of a startup who started moved to hybrid on-prem after six years.
I'm no AWS fanboy, and sure its not the cheapest out there, but this is assuming that all cloud providers provide the same level of services for their machines, which is a complete joke. There are OS, Software, security, ease of tools, even speed of scale considerations that have more to do with why 90% of the internet is on AWS lol. There's a reason the only approved government cloud for years has been AWS. Also, heads up, AWS also offers far more services than ALL of those other providers you mentioned. So when you actually start running a real ONLINE business, and need ETL, MUTLI-AZ, WHAREHOUSEING, AUTOSCALE, S3, ETC, all those things talk to each other out of the box and your vendor list doesn't go from 1 to 50.
Why bother locking in when Kubernetes operators and resources can deploy and maintain the lifecycle of common services? Why would I want some old, forked version of elasticsearch (for example) when I can have the latest version automatically provisioned and scaled with high availability?
What is the AWS value add as Kubernetes operators become mature? They own datacenter space and have contracts for power and internet.
What is the AWS value add as Kubernetes operators become mature? They own datacenter space and have contracts for power and internet.
I think you answered your own question.
> As kubernetes operators become mature
This implies (and I agree) that they’re not mature yet.
Aside from that, there is still more to maintain on kubernetes than on AWS. Those pods don’t patch themselves...
> As kubernetes operators become mature
This implies (and I agree) that they’re not mature yet.
Aside from that, there is still more to maintain on kubernetes than on AWS. Those pods don’t patch themselves...
I agree we are in a transition period and that more maturity is needed for penetration but we are already underway. If you are an executive planning for the next 5 years, this is one of your goals for your engineering organization.
And it's just comparing virtual machines. I don't work at a big tech co -- I'm an enterprise dev working at smaller shops -- and I will try my best to never use virtual machines again.
I want serverless options, I want message queues that scale out of the box, I want stuff that lets me try different k8s configurations.
And I haven't done any research so I guess it's unfair for me to speculate, but will the small guys be guaranteed to be around in 5-10 years? I'm sure Azure, AWS and GCP aren't going anywhere any time soon.
So the comparison means nothing to me. Getting more hosting is way easier than getting new programmers. I'm going the lowest maintenance route I can, provided the costs are somewhat reasonable. I'm willing to pay a premium because stability and easy-to-use features is what I care about.
I want serverless options, I want message queues that scale out of the box, I want stuff that lets me try different k8s configurations.
And I haven't done any research so I guess it's unfair for me to speculate, but will the small guys be guaranteed to be around in 5-10 years? I'm sure Azure, AWS and GCP aren't going anywhere any time soon.
So the comparison means nothing to me. Getting more hosting is way easier than getting new programmers. I'm going the lowest maintenance route I can, provided the costs are somewhat reasonable. I'm willing to pay a premium because stability and easy-to-use features is what I care about.
I think the point is that many AWS customers aren’t doing all of that and still assume they are saving money.
I would finger in the wind guess that a majority of AWS customers are just renting servers and storage and not even scratching the surface of the usefulness of the platform. Just my guess.
I would finger in the wind guess that a majority of AWS customers are just renting servers and storage and not even scratching the surface of the usefulness of the platform. Just my guess.
We don’t assume we are saving money. We are a smallish B2B company. We are far more concerned with being able to scale up when a new client can increase our load (and revenue) noticeably.
The money we save by not having someone babysitting infrastructure and the speed we can move at is well worth it. One client contract basically pays our entire infrastructure bill and then some and we have 30 clients. If we had to hire one Devops person it would be about the same as our yearly bill if not slightly more.
The money we save by not having someone babysitting infrastructure and the speed we can move at is well worth it. One client contract basically pays our entire infrastructure bill and then some and we have 30 clients. If we had to hire one Devops person it would be about the same as our yearly bill if not slightly more.
Sure they aren't, but they don't want it to be a big hassle when it turns out a few years later, they actually need those features.
My employer is okay with paying an extra $10k/mo for stability and predictability.
My employer is okay with paying an extra $10k/mo for stability and predictability.
Yes OVH and DO are much cheaper, I have used both.
I retired this year, last job was engineering manager at Capital One (managed deep learning team, I had little to do with infrastructure). At large scale, AWS is a good choice because of the capable security roles that they support. Not a perfect system, but when a large corporation wants to go all in on cloud deployment, AWS is a reasonable choice.
Personally, I really appreciate getting a free GCP micro instance that usually hosts two semi permanent web demos. Before I bought a good at-home GPU system, GCP was a good option for spinning up instances for machine learning runs, but Azure and AWS is also good for this.
If I were trying to spin up a startup business, I probably would use DO or OVH with a hot smaller backup always running that I could switch DNS settings for rollover - but not nearly as robust as multiple availability zones with one of the expensive cloud providers.
I retired this year, last job was engineering manager at Capital One (managed deep learning team, I had little to do with infrastructure). At large scale, AWS is a good choice because of the capable security roles that they support. Not a perfect system, but when a large corporation wants to go all in on cloud deployment, AWS is a reasonable choice.
Personally, I really appreciate getting a free GCP micro instance that usually hosts two semi permanent web demos. Before I bought a good at-home GPU system, GCP was a good option for spinning up instances for machine learning runs, but Azure and AWS is also good for this.
If I were trying to spin up a startup business, I probably would use DO or OVH with a hot smaller backup always running that I could switch DNS settings for rollover - but not nearly as robust as multiple availability zones with one of the expensive cloud providers.
It is true that AWS is expensive considering cost of raw hardware resources. However, it is also true that it is very easy to create many kinds of apps and their support is not bad. For example using their "serverless" stuff (lambda, api gateway, static hosting in S3 and cloudfront) it is very easy to make apps with hosting cost almost perfectly matching the number of users so the only upfront capital expenditure is dev time. Another thing that is very easy on AWS is micro service apps (geographically distributed or not) consuming their managed services such as elasticache etc. Finally, automated deployments (allowing for continuous development/continuous integration if one wishes to use it) are also easy to implement quickly...
The price one pays for it is vendor lock in. I wrote many mini-non-mission-critical apps to run on AWS, but I would think twice about putting my main business app in AWS without (costly)backup.
I imagine that for startups it might make sense to create their initial product on AWS and when the product takes off spend some of first revenue making it cloud agnostic.
The benefit of AWS is that a startup can implement a proof of concept quickly without having to worry about managing their own elastic search for example, and once it proves viable effort can be added to escape vendor-lock-in.
The price one pays for it is vendor lock in. I wrote many mini-non-mission-critical apps to run on AWS, but I would think twice about putting my main business app in AWS without (costly)backup.
I imagine that for startups it might make sense to create their initial product on AWS and when the product takes off spend some of first revenue making it cloud agnostic.
The benefit of AWS is that a startup can implement a proof of concept quickly without having to worry about managing their own elastic search for example, and once it proves viable effort can be added to escape vendor-lock-in.
You’re always locked into your infrastructure at a certain scale. All the time you waste trying to avoid lock in, you could be using to actually deliver value to customers.
Even if you are just running a bunch of EC2 instances the pain of transitioning your workload and data usually isn’t worth it.
Even if you are just running a bunch of EC2 instances the pain of transitioning your workload and data usually isn’t worth it.
I don't agree with the pricing comparison, it is factually incorrect. For instance the digital ocean equivalent in AWS would not be an EC2 but a similarly sized light sail instance which is $40/monthly flat. There are also other factors like region, on demand vs reservations that are ignored.
Sometimes it's all about priority. I can't speak for the author and I don't know what the author needs, but my experience tells me, again and again, using cloud services is all about productivity, and people tend to underestimate the cost of lost productivity and missing opportunities.
I'll give two examples.
Netflix focused on productivity. They even popularized the phrase "undifferentiated heavy lifting". They were able to build a metric system, in less than a year, that supported full OLAP query with practically arbitrary number of dimensions. The system ingested > 20 billion data points per minute, ran on more than a thousand machines, and did not set quota to any team or service. The entire system was built and operated by merely three engineers. This was simply not possible without AWS. Similarly, Netflix's data pipeline, Elasticsearch cluster, and Druid clusters were developed and operated by two engineers. They were oncall 24x7, yet they rarely got waken up in the middle of a night. You think this was possible if Netflix tried to build its own infrastructure?
The counter example is Uber. Uber was ambitious and wanted to build everything. The results? More than 200 alerts got fired within 2 hours -- the poor oncall engineer couldn't do anything but to keep ack'ing the alerts. Wanted to use Cassandra? 10-page forms that were more tedious than applying for the Ivy schools, weeks of waits, lots of angry emails, countless meetings, and still no Cassandra instances. Wanted to add 50 machines to your ES cluster? PPTs, meetings, escalation all the way to CTO, and weeks of planning. Want to try out Apache Kudu? No way, because our container-based EC2-like system did not support persistent volume, let alone something similar to EBS or EFS. Wanted to use container for your ES cluster? Same answer. Wanted to create your bespoke cluster topology? Same answer because apparently custom network configuration was not supported. After waiting for more than three years? Same answer. If this was not expensive, I don't know what was.
So, you asked if you should save a few thousand bucks a month for your fast-growing company?
I'll give two examples.
Netflix focused on productivity. They even popularized the phrase "undifferentiated heavy lifting". They were able to build a metric system, in less than a year, that supported full OLAP query with practically arbitrary number of dimensions. The system ingested > 20 billion data points per minute, ran on more than a thousand machines, and did not set quota to any team or service. The entire system was built and operated by merely three engineers. This was simply not possible without AWS. Similarly, Netflix's data pipeline, Elasticsearch cluster, and Druid clusters were developed and operated by two engineers. They were oncall 24x7, yet they rarely got waken up in the middle of a night. You think this was possible if Netflix tried to build its own infrastructure?
The counter example is Uber. Uber was ambitious and wanted to build everything. The results? More than 200 alerts got fired within 2 hours -- the poor oncall engineer couldn't do anything but to keep ack'ing the alerts. Wanted to use Cassandra? 10-page forms that were more tedious than applying for the Ivy schools, weeks of waits, lots of angry emails, countless meetings, and still no Cassandra instances. Wanted to add 50 machines to your ES cluster? PPTs, meetings, escalation all the way to CTO, and weeks of planning. Want to try out Apache Kudu? No way, because our container-based EC2-like system did not support persistent volume, let alone something similar to EBS or EFS. Wanted to use container for your ES cluster? Same answer. Wanted to create your bespoke cluster topology? Same answer because apparently custom network configuration was not supported. After waiting for more than three years? Same answer. If this was not expensive, I don't know what was.
So, you asked if you should save a few thousand bucks a month for your fast-growing company?
When comparing virtual machine and ephemeral block storage you're kind of missing the point of AWS "value" for me. It's the deep integration to other services. So yeah DO are perfectly fine if I just need that VPS and a few S3 buckets, but if I need managed Redis? Or something like SSM for managing secrets? Or a managed Elastic search instance?
Redis themselves offer a managed service. There's https://www.envkey.com/ and several alternatives. Elasticsearch offers 3 completely different tiers of support and offerings.
Everyone going to AWS is going there for an AWS-like experience. As someone who runs pretty big workloads on AWS, you still very much need the help when you get to a certain scale, and there is a ton of value in just buying a support contract from Elastic or Redislabs, even when using their managed services or running it on an ec2.
Everyone going to AWS is going there for an AWS-like experience. As someone who runs pretty big workloads on AWS, you still very much need the help when you get to a certain scale, and there is a ton of value in just buying a support contract from Elastic or Redislabs, even when using their managed services or running it on an ec2.
Given that AWS operates at an unprecedented scale yet still seems to charge a premium over smaller operators, can someone explain why their operating margin[0] of ~25% seems relatively small in comparison to what one might expect?
Is this simply because they reinvest a lot of their revenue into product development, or are they much less efficient than they might be able to be?
[0] https://ir.aboutamazon.com/news-releases/news-release-detail...
[0] https://ir.aboutamazon.com/news-releases/news-release-detail...
> their operating margin[0] of ~25% seems relatively small in comparison to what one might expect?
Your setup premise is too constrained, you only provide two possible answers. One, they reinvest a lot; two, they're much less efficient than someone (who?) expects.
The third one is that you're wrong about how much margin is in the business to begin with. The smaller operators are barely surviving at break-even or worse. Companies like Linode and DigitalOcean are not printing large profit margins. DigitalOcean has been a money losing operation (growing via hundreds of millions of dollars in venture capital and debt, plausibly self-sustaining some day when they decide to pull back on spending expansion vs growth... maybe). So the premium that Amazon is charging is precisely what is responsible for them comparatively printing money with a ~25% operating margin. It's not the other way around, such that their margin should be up at 50% if they were run properly as a business. It's something close to remarkable that they have a 25% operating profit margin (and are consistently so profitable), it's the envy of the entire cloud industry.
Your setup premise is too constrained, you only provide two possible answers. One, they reinvest a lot; two, they're much less efficient than someone (who?) expects.
The third one is that you're wrong about how much margin is in the business to begin with. The smaller operators are barely surviving at break-even or worse. Companies like Linode and DigitalOcean are not printing large profit margins. DigitalOcean has been a money losing operation (growing via hundreds of millions of dollars in venture capital and debt, plausibly self-sustaining some day when they decide to pull back on spending expansion vs growth... maybe). So the premium that Amazon is charging is precisely what is responsible for them comparatively printing money with a ~25% operating margin. It's not the other way around, such that their margin should be up at 50% if they were run properly as a business. It's something close to remarkable that they have a 25% operating profit margin (and are consistently so profitable), it's the envy of the entire cloud industry.
Also why they can safely drop prices if someone else eats their lunch.
The effective monthly price for that t3a.large (excluding storage) can vary from $55 to $25 depending on how much you're willing to commit in advance.
See the chart at https://ibb.co/M7nnMvL for some different scenarios.
See the chart at https://ibb.co/M7nnMvL for some different scenarios.
Ignoring that this really isn't an accurate comparison of TCO or performance, this really isn't surprising to anyone who has experience working with enterprise sales. Specs alone will rarely win a sale. Intangibles still have a lot of worth in the b2b space.
Meanwhile I have a 64GB DDR5, 512SSD and 4 TB 7200 RPM hard drive, Ryzen ( 2700 - 8 core @3,2Ghz) for 900$ as a docker host kicking cloud butt.
Indeed, I don't need global-wide services though. But it's more than enough for my localized SAAS.
I'll probably add some PI4's in a cluster and experiment with that ( eg. Hosting 2 message brokers instead of one) or the elastic search that requires 1,5 GB RAM ( I'm not sure yet), since I don't want that in the same server.
Indeed, I don't need global-wide services though. But it's more than enough for my localized SAAS.
I'll probably add some PI4's in a cluster and experiment with that ( eg. Hosting 2 message brokers instead of one) or the elastic search that requires 1,5 GB RAM ( I'm not sure yet), since I don't want that in the same server.
This is an apples and oranges comparison.
You are using commercial-grade hardware, with no ECC RAM and other server-grade features.
You are also missing RAID, the UPS power supply and generator backup, as well as redundant Internet connections to different Tier-1 providers.
You are using commercial-grade hardware, with no ECC RAM and other server-grade features.
You are also missing RAID, the UPS power supply and generator backup, as well as redundant Internet connections to different Tier-1 providers.
I don't care about ECC https://blog.codinghorror.com/to-ecc-or-not-to-ecc/
I had hardware raid 5 before and it failed, because I used similar hard drives ( 3 drives failed within a month, wasn't expecting that).
Working on redundant deployment though ;)
-- My quote still stands as I said it
I had hardware raid 5 before and it failed, because I used similar hard drives ( 3 drives failed within a month, wasn't expecting that).
Working on redundant deployment though ;)
-- My quote still stands as I said it
> I don't care about ECC https://blog.codinghorror.com/to-ecc-or-not-to-ecc/
And that's fine. Though as a counterpoint:
https://louwrentius.com/please-use-zfs-with-ecc-memory.html
> I had hardware raid 5 before and it failed, because I used similar hard drives ( 3 drives failed within a month, wasn't expecting that).
Which goes back to my point. You're comparing your home system, with something like one-9's uptime (90% availability over an entire year) with a cloud provider that is going to give you five-9's or better.
These are not the same things. And you are going to pay more for each additional 9 of uptime.
For your own stuff, yes, of course it can make perfect sense to host it yourself. I've got a personal server running on positively ancient hardware that still manages two-9's availability, and that's fine for my use case. But I wouldn't directly compare that to a cloud provider either.
And that's fine. Though as a counterpoint:
https://louwrentius.com/please-use-zfs-with-ecc-memory.html
> I had hardware raid 5 before and it failed, because I used similar hard drives ( 3 drives failed within a month, wasn't expecting that).
Which goes back to my point. You're comparing your home system, with something like one-9's uptime (90% availability over an entire year) with a cloud provider that is going to give you five-9's or better.
These are not the same things. And you are going to pay more for each additional 9 of uptime.
For your own stuff, yes, of course it can make perfect sense to host it yourself. I've got a personal server running on positively ancient hardware that still manages two-9's availability, and that's fine for my use case. But I wouldn't directly compare that to a cloud provider either.
I said my hardware raid failed, I'm still checking georedundancy / failover at sperate locations.
Ryzen threadripper supports ECC memory actually. And most server grade features you could think about.
Yes, but you still have to buy it.
Server-grade ECC memory is slower and more expensive than what people typically put in their desktop machines.
Server-grade ECC memory is slower and more expensive than what people typically put in their desktop machines.
I've got a 128GB RAM 32 core Threadripper. It purrs.
While I appreciate the analysis, the perspective this whole blog post takes misses the mark on how AWS provides value. You can't look at it simply as providing commodity computation and storage.
As a solo SAAS founder, I use AWS because of all the other services that surround those offerings (Cognito for user management, DynamoDB for database, Serverless stack for API, SES, SQS, Route53, ECS, I could go on for a while) provide a huge amount of value for me in terms of not having to fully set up and maintain that functionality myself. I pay next to nothing for these services and the developer and devops hours saved is worth many thousands of $ to me. However, because of this reliance, I then end up knowingly overpaying for storage, computation, and bandwidth (the latter is by far the biggest concern, the former 2 are fairly minor, so I'm looking at alternative CDN providers). However, other than that, even though my AWS bill is the biggest my SAAS has, I happily pay it. There would be almost no way for me to run this thing solo without it (I'm not saying someone else couldn't, just for me).
As a solo SAAS founder, I use AWS because of all the other services that surround those offerings (Cognito for user management, DynamoDB for database, Serverless stack for API, SES, SQS, Route53, ECS, I could go on for a while) provide a huge amount of value for me in terms of not having to fully set up and maintain that functionality myself. I pay next to nothing for these services and the developer and devops hours saved is worth many thousands of $ to me. However, because of this reliance, I then end up knowingly overpaying for storage, computation, and bandwidth (the latter is by far the biggest concern, the former 2 are fairly minor, so I'm looking at alternative CDN providers). However, other than that, even though my AWS bill is the biggest my SAAS has, I happily pay it. There would be almost no way for me to run this thing solo without it (I'm not saying someone else couldn't, just for me).
This is not an ultra fair comparaison since Amazon has huge discount when you signed up for a year for example.
Not to mention spot instances...
But yes OVH has a lot to offer and got my preference too. But I am French and it is a French success!
But yes OVH has a lot to offer and got my preference too. But I am French and it is a French success!
Yeah this is an extremely limited use case with very few data points. It doesn't take into account corporate discounts, spot instances, reserved instances, savings gained from replatforming/rearchitecting to *aaS options. The author here must have very limited enterprise cloud experience.
Part of this point is that Amazon / Google / Microsoft pricing is intentionally complex, which at the end of the day leads to surprise / higher costs
The premium, and highly profitable, market leader is more expensive than other offerings that are trying to play catch-up on market share. This is not exactly a groundbreaking or unknown finding.
BMWs also cost more then Fiat’s
Also, AWS has become the new IBM and as they used to say “nobody gets fired for choosing IBM.” In the current market if you choose something other than AWS people will question your decision if something goes wrong.
BMWs also cost more then Fiat’s
Also, AWS has become the new IBM and as they used to say “nobody gets fired for choosing IBM.” In the current market if you choose something other than AWS people will question your decision if something goes wrong.
OVH is older than AWS and profitable as it is.
I think this is a nice little piece (though as others have said - not exactly breaking new ground) and I just wanted to add something to this:
> "Smaller providers cannot operate at my yottabyte scale"
I used to work at a company, now closed, that at one point was the #2 or #3 user of S3 in the world. We were not a big company and the fact that we were inexorably bound to Amazon was a harbinger of our doom. That we were too large for most solutions and that we had become trapped on one provider was a sign of our poor decisions and our inability to think laterally and, effectively, do anything other than digging straight down.
I raise this because I think that, if you ever find yourself thinking you're too big for a small provider, I think you should treat that fact as an existential threat to your company. Either you build your own hardware setup or you break up how you do things so you can use smaller companies (even if you chose not to). All the tooling and learning you do to fragment, migrate and validate your data will be invaluable.
> "Smaller providers cannot operate at my yottabyte scale"
I used to work at a company, now closed, that at one point was the #2 or #3 user of S3 in the world. We were not a big company and the fact that we were inexorably bound to Amazon was a harbinger of our doom. That we were too large for most solutions and that we had become trapped on one provider was a sign of our poor decisions and our inability to think laterally and, effectively, do anything other than digging straight down.
I raise this because I think that, if you ever find yourself thinking you're too big for a small provider, I think you should treat that fact as an existential threat to your company. Either you build your own hardware setup or you break up how you do things so you can use smaller companies (even if you chose not to). All the tooling and learning you do to fragment, migrate and validate your data will be invaluable.
Or you just use a cloud provider, which will be able to scale well beyond whatever your company grows to.
> All the tooling and learning you do to fragment, migrate and validate your data will be invaluable.
Too bad this learning has absolutely nothing to do with 99% of any business out there. It adds zero value to the company. In fact, it is a net negative because the company could be doing anything but learning something like that.
Opportunity cost is a real cost.
> All the tooling and learning you do to fragment, migrate and validate your data will be invaluable.
Too bad this learning has absolutely nothing to do with 99% of any business out there. It adds zero value to the company. In fact, it is a net negative because the company could be doing anything but learning something like that.
Opportunity cost is a real cost.
My favorite use case for AWS is being able to spin up temporary resources for rendering, video transcoding, or lab testing, and then shut them back down after a few hours. Likewise I can see a huge win for cyclical traffic loads in only paying for what you use, and in being able to scale up on a dime. Comparing it to a monthly $x/mo provider isn't a fair comparison.
What you're doing is probably much more expensive in the long run than buying a GPU or using a dedicated rendering service.
This isnt a useful comparison, Normal S3 has 6 copies, whereas some of the others have only 3. The Durability of S3 standard is 11 9's
That S3 durability isn't really guaranteed by an SLA, it's just "designed" for 99.999999999% durability, but if you don't get that - too bad, I guess?
To be fair, other providers also don't really offer an SLA on their durability.
To be fair, other providers also don't really offer an SLA on their durability.
My point is that S3 is significantly more durable than the "OVH" type object storage in the posters comparison.
This is not even talked about: One of the biggest cost of cloud is also bandwidth. Usually at about $0.10/GB of outbound transfer.
He includes a pointed link at the end to the Twitter of the AI Dungeon 2 guy - who discovered the hard way that when you have a popular app which involves downloading 6GB every time a new user shows up (or an old user, for that matter), you quickly hit $10k/day bills.
Oh wow- I had not looked at that in detail. TLDR: Guy puts free game online, and is charged $0.30 for every download, raking up a $10k bill.
I immediately disregarded the authors opinion when they put OVH in the same league as major cloud providers. OVH is a source of nothing but spam/malware/outright malicious traffic from our perspective.
We have a job that automatically updates an AWS WAF IP blacklist with their ASN ranges from BGP. We don't want their business or their customers business.
We have a job that automatically updates an AWS WAF IP blacklist with their ASN ranges from BGP. We don't want their business or their customers business.
I think this argument/comparison warrants including the pricing of purchasing a hands on rack from a data center directly.
A full 1Gbps up/down cost about $200/mo, plus ~$150/mo for the managed fiber run. A full locked rack is around 800-1,200$/month. You can load up 40U worth of Dell R730xd, or R740xd (dell outlet). Throw in a few enterprise HP 10g fiber switches (refurbished).
You will quickly start to crush any cloud pricing. But you need IT/VMWare experience.
Keep in mind that a cpu/core on AWS or GCP is a fraction of an actual cpu/core. It’s a spot representation of an x cpu from some arbitrary time way back when. Even if their specs say differently, try running a job on a 2x cpu machine vs 2x cpu on dedicated hardware.
A full 1Gbps up/down cost about $200/mo, plus ~$150/mo for the managed fiber run. A full locked rack is around 800-1,200$/month. You can load up 40U worth of Dell R730xd, or R740xd (dell outlet). Throw in a few enterprise HP 10g fiber switches (refurbished).
You will quickly start to crush any cloud pricing. But you need IT/VMWare experience.
Keep in mind that a cpu/core on AWS or GCP is a fraction of an actual cpu/core. It’s a spot representation of an x cpu from some arbitrary time way back when. Even if their specs say differently, try running a job on a 2x cpu machine vs 2x cpu on dedicated hardware.
Cloud computing is cheaper, not because its actually cheaper, but because of elasticity. If you dont need elasticity, then yes, colo will always be cheaper. If you are in the cloud you should be autoscaling, as that is the main premise of why the cloud took over, and why the cloud can save you money. Only pay for what you need right now
That being said, if you can actually plan well enough in the future for colo, you can also pay 3 years in advanced for aws reserved, and pay significantly less than these prices.
I completely agree. I only mentioned because the article included providers that don’t include the same level of ecosystem benefits, which seems slightly unfair (from a CTO/board decision perspective). I think including a baseline colo option gives insight.
Also, purchasing Dell R7xx hardware, from our experience, has almost 100% uptime at scale, over the past 10+ years, confidently running a 3yr machine 5-7 yrs easily.
Also, purchasing Dell R7xx hardware, from our experience, has almost 100% uptime at scale, over the past 10+ years, confidently running a 3yr machine 5-7 yrs easily.
In some ways you just can't compare, i.e., Cloud Formation and software defined infrastructure is a completely new way of architecting services, but if you're doing more than serving web pages, anything data heavy, the storage and egress costs are incredibly high.
Consider the costs of a rack of Supermicro thumpers (90 bays https://www.supermicro.com/en/products/chassis/4U/946/SC946E...) for long term large online storage vs AWS. Even using Glacier (which is cached tape) is expensive in comparison, especially when read/egress costs are included. Even if you use more enterprisey gear from Dell it is MUCH cheaper to DIY or pay an independent integrator than to use AWS.
AWS itself it doing this for even cheaper than what white box gear sells to retail, because of their huge scale and integration efficiencies. There is a reason AWS makes so much money.
Consider the costs of a rack of Supermicro thumpers (90 bays https://www.supermicro.com/en/products/chassis/4U/946/SC946E...) for long term large online storage vs AWS. Even using Glacier (which is cached tape) is expensive in comparison, especially when read/egress costs are included. Even if you use more enterprisey gear from Dell it is MUCH cheaper to DIY or pay an independent integrator than to use AWS.
AWS itself it doing this for even cheaper than what white box gear sells to retail, because of their huge scale and integration efficiencies. There is a reason AWS makes so much money.
It’s simplistic but there’s value in having such basic comparisons as the offerings differ for more complex use cases. If anything, I imagine basic comparisons will underestimate cost differences once the ancillary charges are taken into account. I can’t be the only one who’s seen organisations surprised by their cloud costs after migration and then struggle to reduce costs.
This couldn't have come at a better time for me. I'm trying to migrate off of Heroku and my default was to AWS, but this definitely gave me more options.
One advantage of AWS is that there seems to often be free AWS credits thrown around. It's hard to compare and quantify that, but something I'm aware of.
One advantage of AWS is that there seems to often be free AWS credits thrown around. It's hard to compare and quantify that, but something I'm aware of.
I've run substantial systems (200+ node operations) on OVH and it was completely unreliable. The hardware kept breaking down, the internal network was terrible, and the support was terrible. Sometimes some backhoe would cut the network connection and cause a 12 hour outage (ok, over 3 years only twice). It was also was inelastic, and provisioning was slow. So if I needed new hardware there could be considerable lead time required.
We also tried Heztner, Linode and others. They were all bad in their own unique ways.
With AWS, and GCP, yes it is more expensive and also have their own set of problems but nowhere near the same scale as these other providers.
Yes if you are running a 1-2 machine setup, who cares. If you are running a professional organisation where you cannot reasonably have downtime without customers screaming, don't go for the cheap stuff.
We also tried Heztner, Linode and others. They were all bad in their own unique ways.
With AWS, and GCP, yes it is more expensive and also have their own set of problems but nowhere near the same scale as these other providers.
Yes if you are running a 1-2 machine setup, who cares. If you are running a professional organisation where you cannot reasonably have downtime without customers screaming, don't go for the cheap stuff.
This - some of remember the MULTIPLE CONTROL PLANE takeovers at these third rate places.
Seriously - root access on the control plane - that was then covered up. You really are going to build your business there?
There is a reason AWS can charge the premium.
I quickly learned my lession in linode at least!
Seriously - root access on the control plane - that was then covered up. You really are going to build your business there?
There is a reason AWS can charge the premium.
I quickly learned my lession in linode at least!
I haven't used OVH a whole lot, but it's my go-to for small stuff because it's such a good value for the price. I have spun up a lot of instances on AWS and OVH now, and haven't noticed a speed difference. In fact OVH seem faster. Are you talking about VM instances or something else that takes a while to provision?
"I do appreciate the youthful incredulity of thinking you can save us a million dollars a year by pointing to a hard-drive rack . It’s good to question the fundamentals! And we’ve done just that. Spreadsheets up the wazoo. We don’t spend a million dollars with glee. But you can try to cut corners on redundancy and availability, and then you can see how much leniency your customers will show you when you lose their files and have to explain yourself. I’m the one who has to say sorry! So I have to be able to look at our setup and believe we did everything possible to keep our customer’s data safe, and then some. This is that."
-DHH
https://m.signalvnoise.com/only-15-of-the-basecamp-operation...
-DHH
https://m.signalvnoise.com/only-15-of-the-basecamp-operation...
Ad pixel servers. Literally go from 1x to 4x at peak. If we had to pay for 4x all the time, it wouldn't be feasible. And then add in that feeding through the managed technologies allows you to build lots of product with small teams means it's a no brainer.
Are there any reasons for going bare metal today? Does it lead to competitive advantage in some scenarios?
Imagine if Netflix or Twitch were founded today - two businesses with demanding media-hosting needs. Would they be better if they used Amazon infra or built out their own data centers and CDNs?
I'm really curious about this because I'd like to get into the space and I'm mulling over bare metal as a competitive advantage vs a barrier to scaling. I've seen how difficult it is to provision and upgrade machines. At the same time, if you were to make it to scale, it seems like having the bare metal and the talent to manage it puts the ball back in your court. Am I wrong?
Imagine if Netflix or Twitch were founded today - two businesses with demanding media-hosting needs. Would they be better if they used Amazon infra or built out their own data centers and CDNs?
I'm really curious about this because I'd like to get into the space and I'm mulling over bare metal as a competitive advantage vs a barrier to scaling. I've seen how difficult it is to provision and upgrade machines. At the same time, if you were to make it to scale, it seems like having the bare metal and the talent to manage it puts the ball back in your court. Am I wrong?
Seems like something you wouldn't want to maintain, especially on day 1. It's always something that can be built and transfer over to after the business has grown and developed.
You mentioned Netflix, they did started with their own data centers eventually they moved their website to AWS, but all the heavy lifting is still happening through their caching servers that they send to ISPs[1].
[1] https://openconnect.netflix.com/en/
[1] https://openconnect.netflix.com/en/
Netflix only uses amazon for the “control plane” - ie the “web app” you’re navigating when using the Netflix app. The actual delivery of media happens via their own CDN (of bare-metal servers.)
The nice thing about choosing Linode is you don’t have to wait for a security incident: https://news.ycombinator.com/item?id=10999397
Good luck trying to get a host with 192GB RAM on OVH. It takes days for the host to be provisioned and you have to fill out forms and provide identification.
On AWS if I want to run a model that requires a lot of RAM (or compute, or whatever) I can have it up and running in minutes without thinking too much about it.
Yes, I pay a premium, but the experience is just superior to anything the value hosting services provide. I find the same to be true for the most part in Azure as well.
I'm happy seeing this comparison, however. It gives me a good sense of just how much of a premium I'm paying. Thanks for doing this.
On AWS if I want to run a model that requires a lot of RAM (or compute, or whatever) I can have it up and running in minutes without thinking too much about it.
Yes, I pay a premium, but the experience is just superior to anything the value hosting services provide. I find the same to be true for the most part in Azure as well.
I'm happy seeing this comparison, however. It gives me a good sense of just how much of a premium I'm paying. Thanks for doing this.
[deleted]
It seems like this is assuming something like a simple webserver without HA requirements, but doesn't explicitly state that assumption. There is no mention of reliability or SLAs.
Hetzner, less than €10:
https://www.hetzner.com/cloud?country=ot
One more idea: "$BIG_TECH has huge developer economies-of-scale".
Whenever I or one of our devs has a problem with AWS or configuration, there's a 99% probability that anyone had this problem before and I find high quality google search results. That's not the case for OVH, Digital Ocean, ...
With developer salaries at 350'000 USD/year, you can now calculate the massive amount of savings.
Whenever I or one of our devs has a problem with AWS or configuration, there's a 99% probability that anyone had this problem before and I find high quality google search results. That's not the case for OVH, Digital Ocean, ...
With developer salaries at 350'000 USD/year, you can now calculate the massive amount of savings.
Your average AWS and Digital Ocean customers have very different needs. Hence different feature sets and different pricing. Running a simple Wordpress blog on AWS is just as stupid as running a 1000 instance cluster on Digital Ocean.
Also, why don't you compare Digital Ocean to Amazon Lightsail? Or spot instances? You can often get those for 1/10 of on-demand price.
Also, why don't you compare Digital Ocean to Amazon Lightsail? Or spot instances? You can often get those for 1/10 of on-demand price.
To some extent, all of these things start looking even more expensive when you realize that you're buying burstable CPU that you aren't allowed to fully utilize. At least with Amazon's t3 instance class, you are assigned "burst credits" which you accumulate by not using the CPU and consume by using the CPU. So you are paying $70 per month for "2 CPUs" but you do not get 5184000 cpu seconds / month, you get significantly less. Digital Ocean, at least, is the same for those cheap instances though I'm not sure how they dole out CPU credits (I think they yolo it and you cross your fingers that you're not sharing a physical machine with the guy doing video transcoding and CI builds).
Both Amazon and Digital Ocean do have dedicated CPU instance types. The prices will make your eyes water. (I have not used the other cloud providers, so I don't know what they're up to.)
Obviously selling CPU on a time-sharing basis is a good idea; most customers aren't maxing out their CPU 24/7, and this lets them pack more customers into fewer machines. (RAM is the killer, though, and you'll see that at every cloud provider, RAM is really what you pay for.) But when you compare these prices to what a CPU costs... it starts to make you think.
My last job really warmed me up to the idea of just running my own servers. All our workloads were containers running in Kubernetes. With Kubernetes, I don't care about individual computers anymore. If one malfunctions, workloads can be easily moved to another computer. All the machine setup work is made machine readable by building containers and authoring Deployment objects, so there is no mental investment in a particular computer or disk image that you get from logging in, installing Debian, screwing around with files in /etc for an hour, etc. Basically, any sort of maintenance involved with physical computers no longer concerned me; repairing a hardware failure basically meant just repairing that hardware failure at my leisure and then moving on.
Combined with this was the fact that we were an ISP, so we had a datacenter, power, networking, and all that the floor below our office. I eventually convinced myself that for a month worth of AWS fees, we could have 10x the computing resources and 10x the bandwidth for a one-time cost. Nobody was motivated to hand over the credit card and let me build the cluster... but for me, running dedicated servers went from "thing that only Google does" to "why isn't everyone doing this!?" It's just not that expensive. And, consumer-grade hardware is really good these days. I drooled over the fast builds we got from a c5.4xlarge AWS instance. A consumer Threadripper build would blow that out of the water for about the cost of 6 months of AWS. (Amazon pays the Intel tax. You don't have to, though.)
Both Amazon and Digital Ocean do have dedicated CPU instance types. The prices will make your eyes water. (I have not used the other cloud providers, so I don't know what they're up to.)
Obviously selling CPU on a time-sharing basis is a good idea; most customers aren't maxing out their CPU 24/7, and this lets them pack more customers into fewer machines. (RAM is the killer, though, and you'll see that at every cloud provider, RAM is really what you pay for.) But when you compare these prices to what a CPU costs... it starts to make you think.
My last job really warmed me up to the idea of just running my own servers. All our workloads were containers running in Kubernetes. With Kubernetes, I don't care about individual computers anymore. If one malfunctions, workloads can be easily moved to another computer. All the machine setup work is made machine readable by building containers and authoring Deployment objects, so there is no mental investment in a particular computer or disk image that you get from logging in, installing Debian, screwing around with files in /etc for an hour, etc. Basically, any sort of maintenance involved with physical computers no longer concerned me; repairing a hardware failure basically meant just repairing that hardware failure at my leisure and then moving on.
Combined with this was the fact that we were an ISP, so we had a datacenter, power, networking, and all that the floor below our office. I eventually convinced myself that for a month worth of AWS fees, we could have 10x the computing resources and 10x the bandwidth for a one-time cost. Nobody was motivated to hand over the credit card and let me build the cluster... but for me, running dedicated servers went from "thing that only Google does" to "why isn't everyone doing this!?" It's just not that expensive. And, consumer-grade hardware is really good these days. I drooled over the fast builds we got from a c5.4xlarge AWS instance. A consumer Threadripper build would blow that out of the water for about the cost of 6 months of AWS. (Amazon pays the Intel tax. You don't have to, though.)
Hey mate, article author here. At work right now so don't have time to read and respond to your post properly but wanted to say thanks for writing the emacs<->quartz bridge at BAML. I found it and brought it back to life and it kept me sane for the 18 months I worked there by allowing me to use emacs instead of QzDev. If you are ever in London I will buy you a pint.
Hey, nice to hear!
It seems like you have a good business idea. At some point people will get off the cloud treadmill and build in house setups. If you create a company that sets up and maintains these at a lower cost than other cloud providers it would get you some business
Ironically AWS also provides that: https://aws.amazon.com/outposts/
Aren't you just describing a lower-priced cloud provider?
You had/have k8 cluster running on top of bare metal servers?
I do, it works. I wouldn’t recommend it unless you’re confident that you can actually manage it.
It’d probably be much easier to run the kubernetes cluster within some virtualization/container setup.
It’d probably be much easier to run the kubernetes cluster within some virtualization/container setup.
Interesting. How much time do you spent operating that thing?
I spent a week making sure I could reliably tear it down and build it up again within minutes, this was by far the hardest part. I ended up hacking around kubespray, but in the future I’ll move to a IPMI-based MAAS setup.
After the initial work to get everything set up it’s been working absolutely trouble free.
After the initial work to get everything set up it’s been working absolutely trouble free.
This is a good start. What would have made this article compelling is a performance vs price comparison. cpu, iops, network, etc. extra credit for sampling a statistically significant number of vms across multiple regions and availability zones per provider.
What I wish is that someone would make a blog post that shows you just how much you can realistically do while staying under the "free" tier limits and what innocent "mistakes" will take you over the line into regrettably expensive monthly bill
Also, I wonder why Amazon hasn't gotten into the ad platform business? They've gotten into everyone else's business. Isn't that the dream - to host a website on their platform and have it be self-supported via ads and possibly even profitable?
I’m a front end developer For work and more and more aws experience is explicitly asked for, I feel like I have no choice but to use their services for personal projects just so I can honestly say I have that experience.
The top comment here rebutts this article from a business perspective:
https://news.ycombinator.com/item?id=21313816
This should include some data transfer and associated costs. Some of the services the services listed include a certain amount of bandwidth in the listed price, while others don't.
News flash: elastic pricing is not optimal for static workloads.
When people choose AWS, they do not need only virtual machines. They need the whole range of products that AWS has to offer, and that range is huge.
I wonder why the author is comparing AWS t3a.large to GCP n2-standard-2. It’s not an apples to apples comparison as t3a runs on AMD while n2 runs on Intel silicon.
You buy a shoe: $50, you buy a shoe exactly the same quality, but it's a nike: $120. Cloud is exactly the same kind of product than anything else
Ironically, the per-terabyte price to buy HDD storage these days is roughly equal to what Amazon charges for object storage per month.
Source: Amazon store ;)
Source: Amazon store ;)
This is not ironic as it's absolutely not relevant.
The HDD that you buy is not redundant, don't provide more than a very limited throughput, is not distributed across multiple geographic areas, is not connected to the internet, comes without any of the hardware to make it spin and is not benefiting from s3 APIs as far as I know.
You are comparing the costs of buying gas vs the costs actually associated with driving.
The HDD that you buy is not redundant, don't provide more than a very limited throughput, is not distributed across multiple geographic areas, is not connected to the internet, comes without any of the hardware to make it spin and is not benefiting from s3 APIs as far as I know.
You are comparing the costs of buying gas vs the costs actually associated with driving.
No one ever got fired for choosing IBM... i mean AWS
It might be CYA in action. However, risk has real cost. Risk of needing more than the little guys provide, risk of them disappearing or the service not keeping up with the times. For some people, the risk way outweighs the extra hosting cost.
Sentence 1: Amazon is probably the largest cloud services company in the world.
Probably? Back button where art thou.
Probably? Back button where art thou.
According to the results, shouldn't the title be "The Microsoft Premium"?
There are lots of apples to oranges here. You can't compare say a Linode or Digital Ocean to an AWS/Azure. One of boxes for experimentation may make sense, I would not run prod though. Also, have you seen the network performance of the smaller providers? (obviously that comes at a cheaper cost too)
In the long run you could save 50% or more if using reserved instances on AWS.
One more idea:
"$BIG_TECH will still exist in 10 or 20 years."
There is no mention of uptime or SLA.
DO is famous for randomly deleting customer data. They probably should not be included on this list.
This entire article is flawed. The true value add of cloud services (AWS, etc) is that you no longer have to pay for disk jockeys to run a data center. To play games with refreshing hundreds or thousands of pieces of bare metal every 3-4 years [EDIT] is to doom your company to crushing O&M.
If you've never done Datacenter operations, then you're missing a giant point that eliminating an entire department of expensive technical people is a huge gain long term.
If you've never done Datacenter operations, then you're missing a giant point that eliminating an entire department of expensive technical people is a huge gain long term.
He is comparing big tech cloud to smaller hosting companies, like DigitalOcean and OVH, not to "run your own machines in your own data centers."
He is talking about virtual machines.
He is talking about virtual machines.
?? All he's saying is that the cloud market leaders are able to charge more. That is, despite Amazon & Microsoft's & Google's economies of scale, there are much lower cost offerings for ordinary virtual machines and storage, all else being equal.
It has nothing to do with on-prem.
It has nothing to do with on-prem.
You didn't read the article at all, did you?
Read it enough to consider his primise was flawed and to judge the article as such. Running other "cloud" providers on-prem is not a recommended unless you're in Gov or Financial. [EDIT] Comparing the other cloud providers is totally a valid exercise. Just disagree with the wording of his/her approach
First, why are we looking at Google's N2 instances rather than N1 instances? The N1 instance would be $52 rather than $61, but the author hasn't talked about why they selected the more expensive N2 instances. Given that competitors don't talk about the CPUs they're using, it seems weird to select the more expensive CPUs for Google.
Second, Google offers dedicated hyper-threads. The author says that cloud providers (like Google) are more expensive, but they've compared dedicated hyper-threads on Google with shared hyper-threads on Digital Ocean. DO offers an 8GB 2 vCPU (2 dedicated hyper-thread) instance for $60/mo, but the author is using the 8GB instance with shared CPU for $40/mo.
So, with storage, Google costs about $52 while DO costs $60 for an equivalent instance. Linode's $60 8GB dedicated vCPU instance does come with 4 cores so it seems like a better value. One can also argue that the included transfer on Digital Ocean offers $8 in value. However, the premium for the instances alone isn't as big for Google. In fact, google seems cheaper.
Now, it's probably a fair comparison for Amazon's instance which only gives you CPU credits for around a third of the time.
Google's automatic sustained-use discounts and dedicated hyper-threads make it really competitive with Digital Ocean (if you don't need a lot of bandwidth) as long as you're doing an apples-to-apples comparison. DO's over-sold CPU instances will be cheaper because you're getting less. If DO didn't offer dedicated hyper-thread instances, maybe you could claim that they were equivalent. But DO literally sells something more equivalent for $60 rather than $40.
EDIT: Also, OVH is a bad comparison. OVH is cheap, but inconsistent in their product offerings. The author notes that they offer hosted DB services, but that's only in France. Their US, Canada, Singapore, Germany, Poland, and Australia locations don't offer that. Their US website is missing pricing on many of the things they seem to offer. Plus, at a certain point, when you're trusting someone with your datastore, you want a high level of certainty. OVH's website doesn't inspire confidence given how much seems like someone didn't even check it. It's certainly cheap, but it seems cheap for a reason.
One final thought: I'm surprised the author hasn't talked about transfer pricing. Cloud providers usually charge a bundle while others charge a lot less. 1 cent per GB at Digital Ocean is a lot less than the 9 cents per GB at AWS. AWS even charges 1 cent per GB for data transfer between AZs in the same region. If you're running a compute and Kafka cluster in 2 AZs, then writing 1GB to a quorum of 3 servers means paying 1.5 cents per GB written (assuming that half the time you're writing to 2 servers outside your AZ and half the time you're writing to one server outside your AZ). Likewise, when you go to read it, if half the time you're not reading from a local AZ, you're paying half-a-cent per GB to read the data. So processing 1GB costs 2 cents. That adds up fast. Everyone compares things like compute instances and seems to ignore data transfer costs. Frankly, Google is competitive on storage and compute costs with places like DO. However, if you are trying to create the next Imgur, you probably care more about transfer costs than compute or storage costs.