Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing(arador.com)
arador.com
Comparing Bandwidth Costs of Amazon, Google and Microsoft Cloud Computing
https://www.arador.com/ridiculous-bandwidth-costs-amazon-google-microsoft/
231 comments
There's a site on Neocities we host for free that would cost $560/mo to host on Amazon's CDN.
It costs me less than $5 with my current configuration (which is also a global CDN). It's way over our "soft limit", but it's an awesome site so I don't care. The important part is, I don't have to care.
This isn't about slightly more expensive tacos. It's about spending $560 on tacos instead of $5. "Meh" wouldn't exactly be my first reaction to getting that bill at the taco cart (or the fanciest taco place in the world, for that matter).
I can get IP transit in datacenters from $240-$600/Gb right now. So even his $960 transit cost for datacenters is off by quite a bit. He's comparing with a pretty high price and it still looks ridiculous.
It costs me less than $5 with my current configuration (which is also a global CDN). It's way over our "soft limit", but it's an awesome site so I don't care. The important part is, I don't have to care.
This isn't about slightly more expensive tacos. It's about spending $560 on tacos instead of $5. "Meh" wouldn't exactly be my first reaction to getting that bill at the taco cart (or the fanciest taco place in the world, for that matter).
I can get IP transit in datacenters from $240-$600/Gb right now. So even his $960 transit cost for datacenters is off by quite a bit. He's comparing with a pretty high price and it still looks ridiculous.
Hi Kyle. You've got me curious now, which Neocity site are you talking about, I would love to check it out.
I look forward to your promised blog posts on how you've affordably built your scalable infrastructure.
Is the basic principle to use cheap cloud VPSes or dedicated hosting (e.g. of the Digital Ocean, Hetzner, or OVH ilk)? And then you can afford lots of servers, and have then geographically distributed?
Is the basic principle to use cheap cloud VPSes or dedicated hosting (e.g. of the Digital Ocean, Hetzner, or OVH ilk)? And then you can afford lots of servers, and have then geographically distributed?
[deleted]
It's the drinks that are expensive, not the tacos.
And soda has a 1,150% markup [1], so it actually is like that.
[1] http://www.businessinsider.com/products-high-markups-2014-7
P.S. You could host it on Github for even less, if you were really price conscious.
And soda has a 1,150% markup [1], so it actually is like that.
[1] http://www.businessinsider.com/products-high-markups-2014-7
P.S. You could host it on Github for even less, if you were really price conscious.
Alright, so $560 for a soda instead of a $5 soda.
> P.S. You could host it on Github for even less.
I'm not going to attempt to save an inconsequential amount of money by hosting a site I'm hosting on another host.
But that strategy probably wouldn't end well. The $560 site uses 25x more bandwidth than Github's fairly low soft limit of 100GB (https://help.github.com/articles/what-is-github-pages/) and is likely above the 1GB site limit (which is also just the sum of any changes ever, because git) of a Github pages site.
From seeing who their CDN provider is (one that charges basically the same CDN rates as AWS), my guess is that Github is paying a lot more for CDN transit than I am. It's perhaps not a coincidence that the 100GB BW limit was quietly introduced after they started using the CDN.
> P.S. You could host it on Github for even less.
I'm not going to attempt to save an inconsequential amount of money by hosting a site I'm hosting on another host.
But that strategy probably wouldn't end well. The $560 site uses 25x more bandwidth than Github's fairly low soft limit of 100GB (https://help.github.com/articles/what-is-github-pages/) and is likely above the 1GB site limit (which is also just the sum of any changes ever, because git) of a Github pages site.
From seeing who their CDN provider is (one that charges basically the same CDN rates as AWS), my guess is that Github is paying a lot more for CDN transit than I am. It's perhaps not a coincidence that the 100GB BW limit was quietly introduced after they started using the CDN.
Yeah Fastly (github's CDN) is ridiculously expensive and they charge for requests just like cloudfront which can become very expensive. They are the only major CDN provider other than cloud services which charges for requests.
We went from cloudfront -> edgecast -> keycdn and our bill dropped from $3000 -> $500 -> $120
Edgecast when you buy is through a reseller is quite cheap but we moved because we needed custom domain SSL which is quite expensive in edgecast
We went from cloudfront -> edgecast -> keycdn and our bill dropped from $3000 -> $500 -> $120
Edgecast when you buy is through a reseller is quite cheap but we moved because we needed custom domain SSL which is quite expensive in edgecast
I realize this is far away from the point, but that infographic starts off stupid. Painting has a 6,000% markup? That could only possibly be true if you don't count the painter's time. Just because it's all labor doesn't mean it's high markup; the painter isn't selling you the paint he's selling you the service.
And mixed drinks are more like a 4x markup. Lower at the high end even. Not sure where they pulled that number from. If it's anything like the coffee total, they'er comparing the garbage ingredients people typically use at home against the highest quality ingredients in the best places.
And mixed drinks are more like a 4x markup. Lower at the high end even. Not sure where they pulled that number from. If it's anything like the coffee total, they'er comparing the garbage ingredients people typically use at home against the highest quality ingredients in the best places.
Agreed. 3,900% for snow removal?
Bandwidth is generally a necessity for cloud hosting, unlike drinks with your tacos.
Lots of bandwidth is generally only necessary for CDN. And that could be anyone.
>There's a site on Neocities we host for free that would cost $560/mo to host on Amazon's CDN.
Neocities isn't just for static websites? How much traffic are getting that it would $560 on s3?
Neocities isn't just for static websites? How much traffic are getting that it would $560 on s3?
I bet that site loses money for Neocities every month. But it's subsidized by the tens of thousands of $5 sites that don't consume their allotment of resources.
Read again: it's free for the user, $5 is the cost to neocities (so yes, it looses them money. $5 a month)
Whoops. You are correct.
This is a nice way to look at the pricing model in the context of something we can all understand: Mexican food.
[deleted]
> I guess I could write a very upset article about how my local mexican restaurant is SERIOUSLY SCREWING ME OVER with their drink prices
I notice that I drink less (too little, even, if I'm out for an evening) because of these prices. If each drink costs a few euros for 200ml glasses, I'd be crazy to drink a liter if at home the cost is somewhere between negligible (tap water) and 1.50 for that whole liter.
What I don't understand is why they can't spread out the profit margin. It'd make me feel a lot better about buying drinks and I'd probably spend more than I do now. Courses are a few euros in ingredients but I don't hesitate to pay €9-15 for it because I know labor is involved and someone has to pay rent. For drinks there is hardly any labor or ingredient cost (assuming all I ask is tap water, which is often since Dutch tap water tastes better than bottled water, and it's not cooled). Prices might also be higher because people don't order it in great quantities, which they might if restaurants wouldn't inflate drink prices to pay for rent (and other such costs), which would be so if you just pay per hour for a seat or something. I heard some places in Italy do that and people were surprised to be charged for it, but it's just itemizing costs and in my opinion more transparent.
I notice that I drink less (too little, even, if I'm out for an evening) because of these prices. If each drink costs a few euros for 200ml glasses, I'd be crazy to drink a liter if at home the cost is somewhere between negligible (tap water) and 1.50 for that whole liter.
What I don't understand is why they can't spread out the profit margin. It'd make me feel a lot better about buying drinks and I'd probably spend more than I do now. Courses are a few euros in ingredients but I don't hesitate to pay €9-15 for it because I know labor is involved and someone has to pay rent. For drinks there is hardly any labor or ingredient cost (assuming all I ask is tap water, which is often since Dutch tap water tastes better than bottled water, and it's not cooled). Prices might also be higher because people don't order it in great quantities, which they might if restaurants wouldn't inflate drink prices to pay for rent (and other such costs), which would be so if you just pay per hour for a seat or something. I heard some places in Italy do that and people were surprised to be charged for it, but it's just itemizing costs and in my opinion more transparent.
People just don't behave rationally to this. Restaurants learned ages ago that people focus on entree pricing, as a result their is little to no margin on many main dishes (sometimes even negative). This is made up on wine, desserts, things that people have price elasticity on or poor information or both.
All that said, the operating margins on high end restaurants are ridiculously low. Franchises and fast food do better at this (economy of scale), but if it is a proper linen-table-cloths and local sourcing sort of place, they are playing with very thin margins at the best of times.
All that said, the operating margins on high end restaurants are ridiculously low. Franchises and fast food do better at this (economy of scale), but if it is a proper linen-table-cloths and local sourcing sort of place, they are playing with very thin margins at the best of times.
That would be workable if humans would be perfectly rational, but they are not. If I set up a pizza shop with totally fair prices across the board, my pizza will cost 8 EUR and the drinks maybe 1 EUR. The pizza shop across the street will however put up a ginormous billboard advertising their 5 EUR pizza with a peace of free pizza bread (and have the 3 EUR drink prices in the small print menu). I get very few customers with my "normalized prices pizza and drinks" billboard. Most people would be just like: WTF? and go to the other shop.
Perhaps an even better analogy is popcorn and soda at the movie theater. Its 10x the price of buying it at costco (perhaps 100x), but you are paying for the movie. In the world of cloud computing this is things like TPUs. I think the lesson here is if you are merely looking for connected bandwidth, maybe cloud providers aren't for you. Shop elsewhere. The point of cloud computing is clearly moving in the direction of data-driven services, at large scale, for big businesses.
Agreed... this article makes a useless one-dimensional comparison of several different services based on their pricing for a single aspect of each of those services. I'm not saying there aren't price discrepancies, or opportunities for more competition that might benefit consumers and enterprises... but the comparisons as portrayed in this article are just not useful except to show how pointless such one-dimensional comparisons are.
As a provider of IaaS Cloud and of dedicated servers and colo, I hear this argument all the time. No one ever seems to include the Network Engineers, monitoring systems, the routers (better have more than 1!), the switches (distribution and access layers), the maintenance, software licenses (where applicable), customer support, cost of IP addresses, Account Payable, ARIN membership, RADB membership, cross-connects, optics, spares and/or support contracts, etc... and finally, you do not use a 1Mbps at 100% for 24hrs per day, so while 1Mbps for a month is ~320GB, in reality, the way most people transfer data, 320GB would look more like 3Mbps at 95th percentile (the way burstable bandwidth is billed)
A basic 1Gbps commit on a 10Gbps port in a data center might cost you from $0.50/Mbps (something like Cogent) to maybe $1.50/Mbps (let's say Level 3), other providers could be $4+/Mbps. By the time you factor in all of the above overhead costs, the true cost of the bandwidth is much much higher on a per Mbps basis.
Don't forget to significantly over-build your stuff, or you might get knocked off-line for anomalies or DoS attacks.
Admittedly, the scale of Google, AWS, Azure makes the cost per Mbps much much lower, but when as others have pointed out, AWS, Google, Azure don't need to charge less than they do.
A basic 1Gbps commit on a 10Gbps port in a data center might cost you from $0.50/Mbps (something like Cogent) to maybe $1.50/Mbps (let's say Level 3), other providers could be $4+/Mbps. By the time you factor in all of the above overhead costs, the true cost of the bandwidth is much much higher on a per Mbps basis.
Don't forget to significantly over-build your stuff, or you might get knocked off-line for anomalies or DoS attacks.
Admittedly, the scale of Google, AWS, Azure makes the cost per Mbps much much lower, but when as others have pointed out, AWS, Google, Azure don't need to charge less than they do.
Straw man argument, you're looking at around $300/month for 20A, 10+ rack units, 100mbit commit to play with in a lot of decent US facilities. That's a few pathetically provisioned VMs on most clouds. I have such a colo for my personal use. This doesn't exclude me from using other providers for DNS and MX slaving, or things like object storage (i.e. I use Tarsnap for DR) and there isn't really any overlap or lost efficiency in mixing and matching those higher level services. My facility's house bandwidth blend provides free DoS mitigation as a nice bonus.
Most facilities and providers don't require you to have your own ARIN registration and block. In fact it's quite difficult to get even the smallest blocks right now. Instead, an ISP will lease you IPs usually at around $1/month.
A pair of x86-64 boxes running OpenBSD can easily virtual switch, route and firewall well above 1gbps. They can announce BGP if needed, if you have a delegation or are muli-homing. A pair of high quality 10g switches may push a few grand on the secondary market but you can ~get away~ with Ubiquiti gear.
This kind of setup can scale up pretty high by dropping the OpenBSD boxes from the forwarding plane and just using them as route reflectors into higher end switches.
There is a slight labor disadvantage that amortizes quickly. i.e. setting up switches and routers isn't hard, and if you aren't sysadmin skilled enough to do that in a few days you probably shouldn't be running VM infrastructure either and retreat to something like Heroku, Lambda or GAE where there is much less room for footshots.
So TL;DR $300/mo and 5k CapEx will run a company, double or triple that for DR of a serious startup. Reap significant saving as a larger company by partnering or outsourcing remote hands and basic ops with a company like yours for the day to day stuff.
Most facilities and providers don't require you to have your own ARIN registration and block. In fact it's quite difficult to get even the smallest blocks right now. Instead, an ISP will lease you IPs usually at around $1/month.
A pair of x86-64 boxes running OpenBSD can easily virtual switch, route and firewall well above 1gbps. They can announce BGP if needed, if you have a delegation or are muli-homing. A pair of high quality 10g switches may push a few grand on the secondary market but you can ~get away~ with Ubiquiti gear.
This kind of setup can scale up pretty high by dropping the OpenBSD boxes from the forwarding plane and just using them as route reflectors into higher end switches.
There is a slight labor disadvantage that amortizes quickly. i.e. setting up switches and routers isn't hard, and if you aren't sysadmin skilled enough to do that in a few days you probably shouldn't be running VM infrastructure either and retreat to something like Heroku, Lambda or GAE where there is much less room for footshots.
So TL;DR $300/mo and 5k CapEx will run a company, double or triple that for DR of a serious startup. Reap significant saving as a larger company by partnering or outsourcing remote hands and basic ops with a company like yours for the day to day stuff.
This is bull. I've used colo and been paying 95th percentile billing for a decade. We've run our own hardware. Much of your list didn't even come in to it. E.g. you don't need your own router, you get awesome ddos mitigation from upstream providers like NTT super cheap. It's been a major competitive advantage to get bandwidth this way.
Cost of IPs? Memberships? Wtf are you talking about? Cross connects? This stuff is all free, not needed or cheap and a one time fee.
Cost of IPs? Memberships? Wtf are you talking about? Cross connects? This stuff is all free, not needed or cheap and a one time fee.
Really depends on your scale. If you just run half a rack of equipment a lot of these costs are factored in by your housing company. But you come to a certain point where you need to invest in a RIPE membership, where you need a router that costs 100k€. Plus a spare one. Where you need someone to watch the monitoring 24h sitting in the DC - just in case. Even if a lot of this stuff is free, it needs to be setup up and managed all the time.
Let alone the cost for DDoS mitigation. Thats an easy 300k€ worth of equipment plus lets say 2x40 GBit/s links for lets say 40k€ each per link and month. Over the course of two years thats easy 1 million euros, just to handle DDoS. Even if you just need 10 GBit/s capacity, you might need 10x that capacity in a DDoS situtation.
Let alone the cost for DDoS mitigation. Thats an easy 300k€ worth of equipment plus lets say 2x40 GBit/s links for lets say 40k€ each per link and month. Over the course of two years thats easy 1 million euros, just to handle DDoS. Even if you just need 10 GBit/s capacity, you might need 10x that capacity in a DDoS situtation.
Here's the reason no one thinks about those things: no one makes their numbers public.
Or at least I've never stumbled across a blog post with anything like a line item cost range for everything that goes into DC or cloud networking.
And in the absence of transparency, yeah, people are going to assume they're getting screwed.
(I understand why this is. The networking free market seems to do a decent job at fulfilling needs, but it turns all those things into secret sauce that shouldn't be shared.)
Or at least I've never stumbled across a blog post with anything like a line item cost range for everything that goes into DC or cloud networking.
And in the absence of transparency, yeah, people are going to assume they're getting screwed.
(I understand why this is. The networking free market seems to do a decent job at fulfilling needs, but it turns all those things into secret sauce that shouldn't be shared.)
Yup. I took a swag at a generalized breakdown of "data transfer" costs https://news.ycombinator.com/item?id=7479030 four years ago. The values might be slightly different today, the logic isnt.
I suppose part of the issue is marketing. The customer reads the bill as "bandwidth." Would consumers understand better if there was a separate "network infrastructure" line item? Would they accept differntial pricing for network quality or class? Dont see why we'd find out as long as theres no downward pricing pressure.
I suppose part of the issue is marketing. The customer reads the bill as "bandwidth." Would consumers understand better if there was a separate "network infrastructure" line item? Would they accept differntial pricing for network quality or class? Dont see why we'd find out as long as theres no downward pricing pressure.
How could I forget electricity: A/B power for a pair of Cisco ASR 9006 routers costs us over $1,000/mo!
The article compares cloud costs to putting a little x86 server on a colo-provided 100 Mbps connection. I think you are not talking about the same kind of service, and also this reference price does include amortized cost of network engineering.
A lot of the things you mention are presumably covered in the cloud provider's service prices for things other than bandwidth.
Another possible explanation is that the ratio of available (or build-out-able) compute power to available bandwidth they have is such that they just can't give you cheap bandwidth and their pricing reflects that.
Another possible explanation is that the ratio of available (or build-out-able) compute power to available bandwidth they have is such that they just can't give you cheap bandwidth and their pricing reflects that.
Cloud data transfer is very overpriced, no doubt about it.
I'm currently paying 0,017 USD for high quality transfer, with a managed network. I could get even lower pricing by switching to 95th-percentile. Maintenance on server hardware is damn cheap with remote hands and hardware warranty.
It would cost me over five times as much to use GCP with 50 TB traffic/month, all things considered.
Cloud providers are cool, but bandwidth is still very overpriced.
Also, your transit prices are way higher that reality, at least for Europe.
I'm currently paying 0,017 USD for high quality transfer, with a managed network. I could get even lower pricing by switching to 95th-percentile. Maintenance on server hardware is damn cheap with remote hands and hardware warranty.
It would cost me over five times as much to use GCP with 50 TB traffic/month, all things considered.
Cloud providers are cool, but bandwidth is still very overpriced.
Also, your transit prices are way higher that reality, at least for Europe.
Actually, we just bought bandwidth for a roll out at Equinix in Munich. $0.50 for Cogent (when added to several other 1G commit on 10G ports in our account. A single 1G commit on a single 10G port would cost more) and we were quoted $1.43 for Level 3 after rejecting a $1.70 quote. Both Cogent and Level 3 were 2yr terms, and in an "on net" location. We are going with another provider besides Level 3 there, but I used these as examples in the parent. You thought it was not "reality", and I refute that.
Router? Gateway? Firewall? Network Access Control? VLANs? Ability to manage all this through declarative version controlled code w/ rollback?
The costs of doing those (well) yourself are not cheap.
Getting them from a provider that's certified to do them well while giving you software control also isn't cheap.
You're comparing cost of gas per gallon to to expense of miles driven per gallon. Pretty sure on your IRS or corporate expense report those aren't the same.
The costs of doing those (well) yourself are not cheap.
Getting them from a provider that's certified to do them well while giving you software control also isn't cheap.
You're comparing cost of gas per gallon to to expense of miles driven per gallon. Pretty sure on your IRS or corporate expense report those aren't the same.
Heck, even the cost of creating and maintaining documentation for the services has to be enormous. The first time I worked with the AWS IoT services I was extremely impressed by the breadth, depth, and quality of the documentation, and it all seemed up-to-date: the screenshots showed the same interfaces I was seeing, and even documentation on small features or use-cases seemed to be accurate and current.
When you put it all together, you're not paying for bandwidth, you're paying for the power and convenience of a robust, high-quality, high-availability solution that provides extraordinary benefits when looked at holistically.
When you put it all together, you're not paying for bandwidth, you're paying for the power and convenience of a robust, high-quality, high-availability solution that provides extraordinary benefits when looked at holistically.
Exactly. Olive Garden is $10 for a pasta plate. I could have bought the ingredients for less than $2.
It takes times and skill to turn the ingredients into something useful.
Amazon, Google, and Microsoft are IaaS/PaaS, not ISPs.
It takes times and skill to turn the ingredients into something useful.
Amazon, Google, and Microsoft are IaaS/PaaS, not ISPs.
Also, employees? These are services that are actively maintained and developed."Alternative 1 – Colocation in a Data Center" doesn't include the 100k/yr you'll have to pay an engineer to manage that hardware.
Most enterprises, you're right. I left people off because you can handle a dozen or more DCs with only a couple engineers if you assume (design for) frequent component failure and only do a site visit/decomm/install once or twice a year.
I have posted a few times about how absurdly expensive all the cloud providers are. If you have a baseline load you should be co-locating bare metal. Any excess capacity you need should spill over into your AWS/GCE/Azure account.
For example: A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
For example: A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
1. Will Dell servers magically appear in your racks all mounted correctly? Shipping delays is a very serious and real problem.
2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time.
3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.
2. Will Dell servers arrived with 0 problems on its components? Any time I ordered meaningful amount of servers, I usually get about 5% fail rate. VMs aren't perfect but you can destroy and recreate in different region almost real time.
3. Ever had to deal with difficult-to-work-with network admin? The cloud is significantly less pain in the ass.
#3 is why we went to Amazon, and its one of the reasons we were successful. Sad as that was. No weeks/months of delays/hassles/fights/sabotage. Just success as fast as we could do our part and figure out the few small hurdles.
...That sounds more like an HR problem than a technical problem.
There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.
There are network engineers in the world who don't live in caves and gnaw at the bones of administrative assistants.
There are, but they are hard to find.
Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else.
I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.
Networking is a great place for assholes to build empires and exert control. In 20 years of professional engagement with distributed and data center networks, most of these guys (and they are almost always guys) running network orgs are an impediment and spend more time helping out their vendor of choice than anything else.
I've run into awesome network guys in position of power only a few times, and they are 100x employees. The last major rollout of a service that I did was literally 3 months ahead of schedule solely because of the efforts of this awesome network dude snowman as recently put in charge.
We saw some good people, but they generally left because they didn't want to work in that environment.
Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.
Eventually replaced with someone helpful, it was amazing what we started to get done. Got lucky there we found the replacement and he stayed long enough to be there when needed.
I wonder if all these 10x network admins are now working at gcp/AWS/Azure.
Oh it was, but it was the situation we had to operate in.
And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
And Amazon allowed to to achieve amazing things that never would have been possible and that's without the idea of things like Kinesis or Redshift that we couldn't have achieved easily/at all even without personnel problems. You simply could never get 3 new servers or drastically respec an existing one in under a day let alone minutes. We were a small shop so it's not like we had spare servers ready to go.
Please send me their resumes..
Is sabotage something you are worried about in general. Just seems like there might be deeper and serious issues if that is a primary concern.
It was sabotage in the 'oops this project isn't working' or 'oops this really import thing came up, no time' or 'didn't I do that?' sense. Having some plausible deniability. Work slowdown/stoppage stuff. Not actively breaking things (mostly). And it was all directed at maintaining control/power.
Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.
Worst person I've ever worked with. When I tell stories at future jobs I fully expect people to not believe me. Hell, people who weren't there for earlier incidents often didn't believer them.
Oh I believe you. I've heard of it before. Was trying to say that if that's an issue, it should be an HR fix more than an infrastructure fix. But I also imagine often that person also convinced management the are an "11x rockstar programmer"...
They were one of the hardest working people in the company. Partially that's because they worked on nonsense projects they made up, and partially that's because they were always putting out fires caused by the fact they did things poorly in the first place.
That's scares me cause I've seen that bit way too many times. There are huddled over the terminal in the evening as all managers leave "Oh look so and so is working so hard" and I am thinking "they are fixing the bug they put in just last week because they didn't bother testing it enough".
Tons of that. Each server was a literal copy of a previous one then modified to fit its new role. So we never got two servers that were the same. They'd have extra junk on them too.
Just basic docker use (or something similar) would have made a HUGE difference. Parts of the department were sent to training but it didn't happen because it would have made him less indispensable.
At one point someone higher up (but not high enough) asked how many servers we had in Amazon and the answer like 10x our physical set.
"And only one of you manages them?"
"On the side, we don't usually have to touch them."
"Then why does it take 4 people for the other servers?"
(Collectively) "Yeah. Why IS that?"
Change control was a big problem too. "Random" changes, things that didn't change at all (even when we could prove it), stuff like that causes SO MANY issues, basically all avoidable.
Just basic docker use (or something similar) would have made a HUGE difference. Parts of the department were sent to training but it didn't happen because it would have made him less indispensable.
At one point someone higher up (but not high enough) asked how many servers we had in Amazon and the answer like 10x our physical set.
"And only one of you manages them?"
"On the side, we don't usually have to touch them."
"Then why does it take 4 people for the other servers?"
(Collectively) "Yeah. Why IS that?"
Change control was a big problem too. "Random" changes, things that didn't change at all (even when we could prove it), stuff like that causes SO MANY issues, basically all avoidable.
A corollary to #1, oops, mistakes were made, these server specs are wrong, I actually need more/less CPU/RAM/storage.
OVH has absurdly low prices on both. If you have a baseline load take a look. Their enterprise offerings are pretty rock solid and they have multiple data centers with direct fiber backplane interconnects.
IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.
IMHO the big cloud providers only make sense if you have a highly elastic load or if you are making extensive use of their as-a-service higher level offerings. If you're running baseline load bare metal destroys them.
I used to use OVH on occasion, but I was really disappointed with their North American-based tech support. Didn't seem particularly plugged-in and English proficiency was shaky. (At this point I'm in AWS because I don't really care about low-point performance and would rather spike up when needed.)
Unfortunately OVH doesn't have any data centers in the US although they seem to be working on them.
Montreal has low pings to the US.
> You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx), and colo it with a 1GB/s blended bandwidth connection billed at the 95th percentile for $150/month.
And then magically maintain it for free?
And then magically maintain it for free?
hopefully with your spare 3437 dollars a month you could figure out how to ever maintain such a thing.
... per server.
It really, really depends on on your use case, but there are some scenarios where bare metal just makes more sense. Also there are ones where cloud makes everything much easier.
North America-oriented B2B app that's a JavaScript SPA witch connects to some lightweight services fronting a database? I think that's a great contender for the cloud.
500-node Hadoop cluster running 24/7 with heavy load? You are going to see some huge savings by maintaining a Datacenter. You can use the extra 1.7MM/mo. to set up some robust networking equipment and have few people on-site.
It really, really depends on on your use case, but there are some scenarios where bare metal just makes more sense. Also there are ones where cloud makes everything much easier.
North America-oriented B2B app that's a JavaScript SPA witch connects to some lightweight services fronting a database? I think that's a great contender for the cloud.
500-node Hadoop cluster running 24/7 with heavy load? You are going to see some huge savings by maintaining a Datacenter. You can use the extra 1.7MM/mo. to set up some robust networking equipment and have few people on-site.
This.
Like every IT trend, "Cloud ops" is the trendy mind-slug that eats everyone's brain and makes them say that everyone who doesn't jump on the bandwagon is going to die broke and unloved in the gutter.
Remember how XML meant massively reduced integration costs and firing all your developers because GUI tools would let business managers connect pretty boxes and then kick back to watch the graphs go up and to the right as business exploded?
Yeah, so AWS is great, if you're building something that works well on AWS. And that's a large class of apps - if you're throwing up a Drupal front end to a CRUD line-of-business backend, need to push notifications and send some email, sure. You're dead in the middle of their target market.
Doing something interesting? Doesn't even have to be as massive as the parent suggests - if you're doing something that blows any of the billable parameters out of the sweet-spot, like bandwidth, storage, latency requirements, things that depend on system-locality, etc., you're much better off building, and using AWS for the pieces which can be broken off that don't hit the pain points.
Like every IT trend, "Cloud ops" is the trendy mind-slug that eats everyone's brain and makes them say that everyone who doesn't jump on the bandwagon is going to die broke and unloved in the gutter.
Remember how XML meant massively reduced integration costs and firing all your developers because GUI tools would let business managers connect pretty boxes and then kick back to watch the graphs go up and to the right as business exploded?
Yeah, so AWS is great, if you're building something that works well on AWS. And that's a large class of apps - if you're throwing up a Drupal front end to a CRUD line-of-business backend, need to push notifications and send some email, sure. You're dead in the middle of their target market.
Doing something interesting? Doesn't even have to be as massive as the parent suggests - if you're doing something that blows any of the billable parameters out of the sweet-spot, like bandwidth, storage, latency requirements, things that depend on system-locality, etc., you're much better off building, and using AWS for the pieces which can be broken off that don't hit the pain points.
Correct!!!
Where can I find competent sysadmins capable of managing a system like that on their own for $40,000 a year?
Doesn't make much sense for one server... but if you have 5, then you are looking at $200k. Then as you go up the numbers start making more sense.
That might get you one admin (we office workers have a mistaken belief that our salary is most of our cost. It's somewhere just north of half.).
What do you do when they're on vacation/sick/sleeping?
Thing is, I really want us to host our own stuff. For one thing if you don't host anything you lose all competence and then you really are beholden to your cloud provider.
But it seems to be that we are in this situation until someone figures out how to comodify servers - and server orchestration - to the point where this stuff gets cheaper to manage again.
This pendulum always swings. No sooner will we control our own hardware again than someone will come up with a new way to centralize it.
What do you do when they're on vacation/sick/sleeping?
Thing is, I really want us to host our own stuff. For one thing if you don't host anything you lose all competence and then you really are beholden to your cloud provider.
But it seems to be that we are in this situation until someone figures out how to comodify servers - and server orchestration - to the point where this stuff gets cheaper to manage again.
This pendulum always swings. No sooner will we control our own hardware again than someone will come up with a new way to centralize it.
Totally, the thing that's super cool about the amazon instances you don't have to do anything with users, authentication, authorization, networking, dns, backups, monitoring, logging, firewall rules, security patches and upgrades.
With a bare metal server you have to manage all of that.
With a bare metal server you have to manage all of that.
Oh my goodness man, have you managed Amazon instances? Setting up VPCs, IGW, IAM, DNS, EBS Snapshots (and deletions until recent lifecycle), custom metric in coudwatch, and Security Groups is a seriously challenging task just to learn the basics, much less do it right, much less automate it.
It's not a bed of roses in the physical world either, but you're simply wrong to say it's easy in the "Cloud" and hard in a DC/Bare Metal.
It's not a bed of roses in the physical world either, but you're simply wrong to say it's easy in the "Cloud" and hard in a DC/Bare Metal.
(it was a joke.) There are a zillion things to think about if you're going to have even one machine public on the internet, regardless of the hosting solution. At a small organization there's always that one programmer that's like half sysadmin.
I find it amusing that people pretend without bare metal, you'll get all of that person's time back.
I find it amusing that people pretend without bare metal, you'll get all of that person's time back.
I missed it. Thanks.
It's not as if using AWS etc eliminates the need for a good sysadmin...
You can buy sysadmin services. This isn't a full time job at that level.
Then you are just trading one fee for another, but the "sysadmin services" you are ordering most likely has slower response time, less resources, less knowledge, and worse uptime guarantees than someone like aws.
Yeah, at the larger sizes it might make sense, but not for everyone.
Yeah, at the larger sizes it might make sense, but not for everyone.
Nope you have a number for the person that provides the service and you can call her/him. Unless you are at several million per month spend at AWS have fun getting a hold of someone when sh#t hits the fan.
At smaller sizes too. If you just have one server, it's unlikely that the slower response time will loose you 3000 dollars per month which is the savings of dedicated service versus aws. Many small businesses run on leased hardware with on site support. Rackspace for instance made their fortune in that sector.
Well you're not comparing to AWS without ops, you're comparing to hiring someone for non-AWS vs managing AWS on your own. Outsourced ops know your systems. They have contracted guarantees. They can manage your own systems, hosted systems or even AWS. Using not-AWS saves you enough to pay for that.
> Outsourced ops know your systems.
I worked for a MSP for many years, and was well acquainted with our regional competitors. Our uptime and theirs was nowhere near AWS-level.
You're getting a different engineer every time you pick up the phone. They most certainly do not know your systems like an internal team would. I regularly saw cascading/circular issues caused by lack of familiarity and/or poor change management.
Owning and administering your own iron makes sense at a certain scale, but it's a much bigger scale than most companies will reach.
I worked for a MSP for many years, and was well acquainted with our regional competitors. Our uptime and theirs was nowhere near AWS-level.
You're getting a different engineer every time you pick up the phone. They most certainly do not know your systems like an internal team would. I regularly saw cascading/circular issues caused by lack of familiarity and/or poor change management.
Owning and administering your own iron makes sense at a certain scale, but it's a much bigger scale than most companies will reach.
[deleted]
That means you only have about $40k/yr to hire a sysadmin assuming they need no supplies or tools...
Can probably hire a someone at minimum wage to watch it 24/7 in case it spontaneously combusts.
Probably combusted because he plugged the 230v power supply to the 600v plug.
Minimum wage => You get what you pay for.
Minimum wage => You get what you pay for.
you can also pickup an enterprise flash device and have a few terabytes of 500k+IOPs for a few grand whereas reaching a similar performance profile via provisioned IOPs will cost you 10x as much every month.
You're comparing apples to oranges -- if you want a couple TB of locally attached storage, you'd get an i3.2xlarge for $450/month with 1.9TB of NVMe SSD (plus 8 cores CPU + 60GB RAM).
Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
Using EBS volumes with provisioned IOPS give you some advantages that you don't get with a bare SSD -- like snapshot support and the ability to easily detach it and move it to another server.
you're right, the performance of the local storage on i3 instances is a lot better than normal EBS but still nothing close to what you would get from just buying an Intel P4500 or similar though.
http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...
http://www.aerospike.com/benchmarks/amazon-ec2-i3-performanc...
Double the P4500. The raid is not gonna store data out of thin air.
And add a third one while you're at it.
Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.
Given the price of the server and the data you have on it, it's important enough to warrant a hot spare.
I'm not sure which region you priced, but an m4.16xlarge in us-east-1 (Virginia) is $3.20/hour, or $2304/month.
But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want).
Still not as cheap as owning the server, but not quite as bad.
But if you are comparing it to a purchased server, you ought to be looking at Reserve pricing, a 1 year reserved instance is $16K/year, 3 years is $32K (or you can pay monthly if you want).
Still not as cheap as owning the server, but not quite as bad.
I had selected a dedicated instance. It's been my experience the difference in performance between dedicated and non dedicated is dramatic.
edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.
edit: I had selected a dedicated instance in order to compare apples to apples. By definition you don't share a colo server.
With an m4.16xl, there's no effective difference between a dedicated or default tenancy instance - only one .16xl can run on a physical hypervisor, which is reflected in the pricing, it's $3.20/hour whether it's dedicated or not:
https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...
For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default.
There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running)
We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.
https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...
For smaller instances, there is around a 10% increase for dedicated, for example, for an m4.4xl, the price goes to $0.88 for dedicated tenancy versus $0.80 for default.
There is a $2/hour fee for each region in which you have dedicated instances, but that gets lost in the noise if you're using significant AWS resources. (it's $2/hour regardless of how many instances you're running)
We haven't noticed any significant difference between performance on dedicated and non-dedicated instances, and only use dedicated instances where required for compliance/regulatory reasons.
Why get a dedicated instance and not get reserve pricing? You can get those instances for a fraction of what you're currently paying
It's not the hardware you're paying for, it's the services. How much does it cost to employ a systems administrator? A lab tech? And so on. You see exactly the same pattern in software services. People pay more to buy products that are "turn key" versus having to hire someone to do tons of work building out systems or services or what-have-you.
Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.
Also, hiring people is a risk, and it doesn't happen instantly. When you have a gap of months without a sysadmin things are bad. Hiring a bad sysadmin is even worse. Not to mention the impossibility of hiring 1/2 or 3/4 of a sysadmin. There's scaling limits there. If you have the resources to DIY, that can be a key competitive advantage, but not everyone does.
Often, you still need a "systems administrator" with the cloud. Except, now he calls himself a DevOps guy and charges a lot more. ;) AWS isn't going to configure itself.
Use google cloud or softlayer and the same machine is $1500 a month.
It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.
It's no secret that AWS is a rip off on instance pricing. Use the competitors if you worry so much about pricing.
[deleted]
How does that work out? Looking at the per-hour figures in AWS and GCE, they're similar for a 64-core, quarter terabyte RAM machine ($3.20 AWS, $3.40 GCE). Both vendors have ways to get your bill discounted (with google's being easier). AWS's instance pricing has been fairly comparable to other vendors for a couple of years now; it's their bandwidth that's costly.
What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
What are most people running anyway? Using this kind of behemoth server as a 'typical' example seems out of line.
Google n1-standard-64 => $1710
AWS m4.16xlarge => $2592
Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month.
That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box.
On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
AWS m4.16xlarge => $2592
Google base is slightly lower and it gives you 30% off automatically on instances that run 24/7 for the month.
That's typical of on premise deployments. You get big servers because they are so annoying to deploy you might as well fill everything possible in the box.
On the clouds, you would create a group of VM per service, with appropriate size for the service. It is significantly more efficient.
Isn't that $1710 already including the 30% discount? (It works out to $3.4 x 30ish days x 70%). You get similar discounts on AWS, but you have to commit for a year - given the OP's intent of leasing a Dell server for three years, this shouldn't be dismissed.
> A dedicated m4.16xLarge EC2 instance in AWS is $3987/month. You could build that same server for $15,000 through Dell, lease it at $400/month (OpEx)
Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times.
Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)
Cherry-picking a bit there. You're entering a 3-year commitment with the Dell lease, but not with the EC2 lease. Take a 3-year RI on that EC2 instance, and the price drops considerably. If you also take away the 'dedicated' requirement, the 3-year RI is about twice as expensive, not ten times.
Edit: correction from Johnny555 above: this top-tier server doesn't have a 'dedicated' option, so there's not an 'extra' cost for that. It's just twice the price, if you do a 3-year commitment (as you are doing for the dell)
You're paying for the staff time to set up the lower-level infrastructure and keep it running at _least_ as reliably as if you had your own staff on bare metal, quite likely more so. (including scaling it when you need more etc).
It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do.
I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.
It is interesting that they end up charging you this for _bandwidth_ though. Their staff time doesn't actually probably scale with bandwidth, but their prices do.
I think it's probably kind of a way of pricing based on a proxy for customer size, likely revenue, and ability/willingness to pay. Rather than as a % margin on actual costs.
When we account for cost of,
1) remote hands
2) network switches
3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate.
4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload.
5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay?
I'm not sure it becomes a simple price of server + colo cost comparison.
With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc.
This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles
However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.
1) remote hands
2) network switches
3) hardware components you'll need for redundancy, I've personally replaced network cards because fans burned out or HDDs because of usual failure rate.
4) when I'm done using a n1-highcpu-32 I stop using it and not pay for it, or when I need 5 of those for 2 hours I provision them and stop when I'm done running a workload.
5) upgrades to hardware or additional hardware, intel released skylake, should I buy those and try to sell my Haswell on eBay?
I'm not sure it becomes a simple price of server + colo cost comparison.
With GCP (I use daily), AWS (I used to use) or Azure, what I'm paying for is someone else to focus on things like colo leases, capacity expansion, redundancy, physical security and provide higher order services - load balancers, BigTable/BigQuery, CloudSQL/RDS, GKE, DataProc.
This allows me to run all of infra as close to capacity as possible while avoiding any wasted cycles
However I do miss going to data centers and hear my machines hum and blink, that is quite the feeling.
I just landed a few racks of bladecenters and storage. High 7 figure deal. This was probably the best implementation project ever... everything ran on time, the vendor delivered everything on time, facilities was 100% ready and pre-cabled, etc. I'm incredibly proud of our team -- our biggest issue was someone ordered the wrong SFP. Butter.
Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing.
That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.
Timeline: 120 days procurement (including product evaluation and PoC), 30 days install activity, 10 days buildout. 10 days software build, 15 days testing.
That's a little over 6 months. A ton of vendor/build engineering/tester time. And we'll be at 50% capacity for another 4 months. Or... we could spin the whole thing up on AWS/GCE/Azure in < 30 days and only consume what we need when we need it.
Which hardware vendor did you go with?
This analysis is way too oversimplified. It completely ignores the shape of the traffic (real apps have peaks and valleys of usage - they don't pump exactly 100 Mbps every second of every day). Cloud providers charge the same amount regardless of how bursty your app is, and they have to provision capacity so that all customers get good performance even under unusual spikes (the more spiky your traffic, the better a deal per-MB pricing is for you). And of course it ignores all the ancillary networking HW and SW that supports these services, and all the labor you save by not having to manage that stuff yourself.
I've analyzed the cost of cloud services to death (I've worked for a couple of them) and the only way they aren't great deals is if you don't need high quality operations (i.e. if you can deal with slow-downs or occasional outages then you can do better elsewhere). Otherwise, if you're small-scale then these marginal cost differences don't matter, and if you're larger scale then call up these cloud providers and get yourself a discount off the list price.
I've analyzed the cost of cloud services to death (I've worked for a couple of them) and the only way they aren't great deals is if you don't need high quality operations (i.e. if you can deal with slow-downs or occasional outages then you can do better elsewhere). Otherwise, if you're small-scale then these marginal cost differences don't matter, and if you're larger scale then call up these cloud providers and get yourself a discount off the list price.
(Bandwidth is ambiguous in this context so I'll use "data transfers" instead)
I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
Customers with transfer-heavy applications will always buy their servers from providers with unlimited transfers like OVH[0][1], where you can do hundreds of terabytes a month with no extra charges (1.5 Gbps * 3600 * 24 * 30 = 486 TB). Even if AmaGoogSoft lowered their transfer prices by 100 fold their pricing still can't compete with OVH.
Companies with enough engineering resources can always go with the best of both worlds: transfer-heavy servers on OVH, and "regular" servers on AmaGoogSoft. The expensive data transfers will only hit smaller outfits, but these customers won't switch because it's not worth the hassle to split your hosting across two providers.
[0] https://www.ovh.com/us/private-cloud/options/bandwidth.xml
[1] https://www.ovh.co.uk/web-hosting/unlimited_traffic
I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
Customers with transfer-heavy applications will always buy their servers from providers with unlimited transfers like OVH[0][1], where you can do hundreds of terabytes a month with no extra charges (1.5 Gbps * 3600 * 24 * 30 = 486 TB). Even if AmaGoogSoft lowered their transfer prices by 100 fold their pricing still can't compete with OVH.
Companies with enough engineering resources can always go with the best of both worlds: transfer-heavy servers on OVH, and "regular" servers on AmaGoogSoft. The expensive data transfers will only hit smaller outfits, but these customers won't switch because it's not worth the hassle to split your hosting across two providers.
[0] https://www.ovh.com/us/private-cloud/options/bandwidth.xml
[1] https://www.ovh.co.uk/web-hosting/unlimited_traffic
> I personally don't see the outrage. AmaGoogSoft overcharges for data transfers because they know they can get away with it and that lowering it won't attract more customers.
I know a handful of scientists, myself included, who would consider cloud computing were it not for the expensive egress costs. The ease of spinning up lots of computing power for scientific modelling is useless if retrieving the vast quantities of raw output data is costly. I will have to investigate OVH though as a potential opportunity--thanks.
I know a handful of scientists, myself included, who would consider cloud computing were it not for the expensive egress costs. The ease of spinning up lots of computing power for scientific modelling is useless if retrieving the vast quantities of raw output data is costly. I will have to investigate OVH though as a potential opportunity--thanks.
The cloud pricing model has, I would say, definitely been set for ecommerce uses, not scientific computing. Ecommerce uses use a lot less data, and if they are using a huge amount of data they are likely to have a corresponding amount of revenue to pay the high prices.
Scientific computing -- you've probably got a ton of data, and your 'revenue' from project-based grants doesn't really scale with the amount of data and other computing resource needs you have, at least not to the extent it does for ecommerce.
Possibly AWS could make money on scientific data at prices you could afford... but they probably make the _most_ money by targeting their pricing model at ecommerce, which has the most money to spend.
Possibly there's a business model for a cloud-provider focusing on scientific computing. I mean, there probably are such providers already I don't know about? Maybe? But it'd be a tough business, the people paying are pretty budgetarily constrained, their budgets don't neccesarily go proportionaly as their resource needs go up, they traditionally are (ironically) averse to innovation in IT, there might not be enough likely customers to give the provider the scale they need to make money, etc.
Capitalism!
Scientific computing -- you've probably got a ton of data, and your 'revenue' from project-based grants doesn't really scale with the amount of data and other computing resource needs you have, at least not to the extent it does for ecommerce.
Possibly AWS could make money on scientific data at prices you could afford... but they probably make the _most_ money by targeting their pricing model at ecommerce, which has the most money to spend.
Possibly there's a business model for a cloud-provider focusing on scientific computing. I mean, there probably are such providers already I don't know about? Maybe? But it'd be a tough business, the people paying are pretty budgetarily constrained, their budgets don't neccesarily go proportionaly as their resource needs go up, they traditionally are (ironically) averse to innovation in IT, there might not be enough likely customers to give the provider the scale they need to make money, etc.
Capitalism!
Yeah I suppose my argument is a bit moot given Amazon probably makes plenty of money in the private sector without having to worry about academia. But if they could get the cost down to be on the level of buying and running your own hardware, they could see an emergent market.
In fact, the federal government in its current business-oriented mindset may even lean towards a contractor handled machine to replace ones like Yellowstone at NCAR or those at NOAA NCEP if the price was right.
In fact, the federal government in its current business-oriented mindset may even lean towards a contractor handled machine to replace ones like Yellowstone at NCAR or those at NOAA NCEP if the price was right.
It would make a lot of sense if grants covered such costs, even at current AWS prices.
Currently, I suspect a lot of IT-heavy grant-funded projects are basically relying on university infrastructure being provided without full (or in some cases any) project-based cost recoup, which universities are increasingly loathe to do. The indirect/overhead margin probably doesn't always really cover it. (And it's not like a university is going to let you use 'overhead' dollars for AWS, _even if_ it would actually be a cost savings compared to the local IT you are using instead!)
It's not really rational economic decisions being made all around.
Proper professional IT is expensive. People still think automating is supposed to save them money, but turns out, nope. Many industries are still relying on unprofessionally provided IT instead of paying for professional IT.
Currently, I suspect a lot of IT-heavy grant-funded projects are basically relying on university infrastructure being provided without full (or in some cases any) project-based cost recoup, which universities are increasingly loathe to do. The indirect/overhead margin probably doesn't always really cover it. (And it's not like a university is going to let you use 'overhead' dollars for AWS, _even if_ it would actually be a cost savings compared to the local IT you are using instead!)
It's not really rational economic decisions being made all around.
Proper professional IT is expensive. People still think automating is supposed to save them money, but turns out, nope. Many industries are still relying on unprofessionally provided IT instead of paying for professional IT.
If you're an academic you could probably negotiate with google/msft/aws for a discounted if not free egress rate.
May I ask why you want to retrieve it? Wouldn't you just leave it there, and download only the conclusions?
I can think of a few off the cuff.
You're likely not running a model just for yourself. You have collaborators at other institutions that need data to compare or combine with something else.
Your grant may require you distribute the output for use by other researchers. That could either entail being hosted by you or by an agency or other entity. But you still have to get the data to them.
A reviewer for your publications may request the data.
That brings up another point that the review process can be upwards of a year.
Oftentimes you have to go through an exploratory data analysis, where you don't even know what your final analysis will entail.
You're likely not running a model just for yourself. You have collaborators at other institutions that need data to compare or combine with something else.
Your grant may require you distribute the output for use by other researchers. That could either entail being hosted by you or by an agency or other entity. But you still have to get the data to them.
A reviewer for your publications may request the data.
That brings up another point that the review process can be upwards of a year.
Oftentimes you have to go through an exploratory data analysis, where you don't even know what your final analysis will entail.
[deleted]
How much raw output data are we talking about? What order of magnitude? Unless we're talking PB a month EC2 prices seem fairly reasonable. Also keep in mind intra-region communication between instances is free
How is this is a surprise to anyone? The big players are all pushing their clouds because its a cash bonanza. It's the SaaS model for hardware, make money forever because your customers never own anything.
I've done the math many times and it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
I've done the math many times and it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
Theres other hidden costs as well.
What happens when a harddrive fails in your colo? Unless you have best practice backups, you will lose customer data and trust.
GCloud abstracts the issue of hardware so you can focus on real business development. And to some, that's worth the cost until they can properly afford all the hidden costs of independent hosting.
What happens when a harddrive fails in your colo? Unless you have best practice backups, you will lose customer data and trust.
GCloud abstracts the issue of hardware so you can focus on real business development. And to some, that's worth the cost until they can properly afford all the hidden costs of independent hosting.
These are solved problems though. RAID works great around 99% of the time and for the others you can use off-site backup. It's extremely rare that something needs a true 100% uptime. Salesforce and Reddit still go down for maintenance windows in 2017. For one project I setup log streaming with Postgres to the cloud(since inbound BW is free :) ) and ran a hot backup there just in case.
Really though in my experience the cloud is far less reliable than colocation. I've had AWS VMs "degrade" or randomly die countless times but I can't remember the last time my Colo boxes went down, probably not since I last upgraded the OS.
Really though in my experience the cloud is far less reliable than colocation. I've had AWS VMs "degrade" or randomly die countless times but I can't remember the last time my Colo boxes went down, probably not since I last upgraded the OS.
> it's orders of magnitude cheaper to colocate as long as you can afford an IT guy and the upfront cost of hardware.
It may be. But for many people, it does not matter.
Let's say you work at the place (I do) where "compute" cost is like ~0.5% of the total costs. Almost all of the cost is employee's salaries and benefits.
BTW we get (and from what I hear, everyone does) offers from other cloud vendors to switch, and get 1 year free or whatever. And it gets ignored because it is a big risk for a tiny possible gain.
It may be. But for many people, it does not matter.
Let's say you work at the place (I do) where "compute" cost is like ~0.5% of the total costs. Almost all of the cost is employee's salaries and benefits.
BTW we get (and from what I hear, everyone does) offers from other cloud vendors to switch, and get 1 year free or whatever. And it gets ignored because it is a big risk for a tiny possible gain.
Fair enough. I just think that infastructure is a bigger differentiator of success than most give it credit for. By using the cloud you're giving up the possible performance and money advantages of doing it yourself.
It's really just a new form of vertical integration. If you're working only at the high levels you're not only paying someone to do that rest for you, those people are making profit off of you. Profit that could be yours.
It's really just a new form of vertical integration. If you're working only at the high levels you're not only paying someone to do that rest for you, those people are making profit off of you. Profit that could be yours.
Worth noting that Digital Ocean doesn't actually bill for bandwidth. They say they do in their Droplet template descriptions, but they really don't. I've pushed many many terabytes to/from my Droplets and never received a bill for it. But you need to cap your individual Droplet bandwidth using something like tc[0] around ~400MB/s or they'll shut off the network interface (DDoS detection).
[0] http://tldp.org/HOWTO/Traffic-Control-HOWTO/intro.html
[0] http://tldp.org/HOWTO/Traffic-Control-HOWTO/intro.html
indeed. I'm surprised how buried this comment is, because it's a really good deal (until they change things). anecdotally, DO sent me an email when one of my droplets was sustaining around 200MB/s, but it wasn't a "knock it off" email, it was an "is this intentional" email. is the ~400MB/s figure from your own experience?
The bandwidth is the soda: https://news.ycombinator.com/item?id=12270129
I dont think its fair to compare GCP's egress costs to a colo's. A collocation is simply sending your packets straight to the internet, where-as GCP routes your packets over private fiber to its closet POP to your user. Giving you better latency.
This is only true with Google as far as I know. And I'm fairly sure that all cloud providers only have maybe 10 edge routers in the US so the utility is limited. You can't have too many edge routers because they rely on IP Anycast which is dangerous to do if your routes are too similar.
[deleted]
You are right, the Google Network magic is awesome. Yet I agree with the opinion here. They charge more than they should. Yet I think in the long run it will become cheaper (AWS is always pushing for price reduction and the others will follow)
Isn't GCP the one that pushes for price reductions and then the rest will follow?
[deleted]
The last time I measured AWS latency to anywhere, it tens of milliseconds higher than the super-cheap-and-crappy virtual server service I compared it to.
You can measure it now:
- http://www.gcping.com/
(From your browser to multiple Google zones)
- http://www.gcping.com/
(From your browser to multiple Google zones)
Back of the envelope calculation:
- assume $100/TB for cloud data transfer
- assume one employee full time equivalent to manage colo'd servers ($10,000/mo), plus $30/TB data transfer
The break even point for the colo'd setup from a networking perspective is:
Weaknesses of this model:
- Data transfer only. Depending on what else you're doing you could also save a lot on compute and storage.
- I don't know that much about how colocated data transfer would be priced ... i.e. do you need overprovision to guarantee availability, etc.
- one employee to handle servers to replicate the Amazon AWS experience ... could be highly variable depending on what AWS features you are using.
- assume $100/TB for cloud data transfer
- assume one employee full time equivalent to manage colo'd servers ($10,000/mo), plus $30/TB data transfer
The break even point for the colo'd setup from a networking perspective is:
10,000 + 30X = 100X
X = 10,000/70 = 142 (TB/month)
Weaknesses of this model:
- Data transfer only. Depending on what else you're doing you could also save a lot on compute and storage.
- I don't know that much about how colocated data transfer would be priced ... i.e. do you need overprovision to guarantee availability, etc.
- one employee to handle servers to replicate the Amazon AWS experience ... could be highly variable depending on what AWS features you are using.
This calculation is an all or nothing affair though. The reality is that you can avoid most AWS bandwidth charges by direct connecting to an internet exchange. It's still only worth the effort if you're looking at a 6 figure bandwidth bill, but it doesn't require new employees.
I guess what I'd like to know is: at what point is it worth seriously evaluating alternatives to just using cloud services and paying the sticker price -- which means evaluating the labor cost as well as the cost of the service, where this submission and many others I've seen will quantify the cost of the services and say that bare metal is so much cheaper while ignoring the cost of managing it.
The general break even point is circa 200k USD per month, assuming the company has spare capital, and the ability to commit to using the same gear for 3 years.
It's useful to keep in mind that in this type of scenario the goal is not to replace a cloud provider but to leverage capital to get better bang for the buck on select services. ex. using in-house compute farms and leveraging s3 for storage
It's useful to keep in mind that in this type of scenario the goal is not to replace a cloud provider but to leverage capital to get better bang for the buck on select services. ex. using in-house compute farms and leveraging s3 for storage
50 rps is a very modest box you obviously will want at least 2 but at any rate the difference between that and AWS is not a full time employee it's more like 10h per month. Why would colo data transfer cost $30/TB :)?
The $30 number comes from the original article -- If I were to make the colo data transfer free and change nothing else, break even point is 100TB/mo.
For the employee cost I agree if you just have a couple boxes pumping out data your labor cost will be quite low. I'm imagining what else you might have happening on your back end using AWS services that you'd also have to replicate in your colocated installation to preserve the efficiency, which would take labor. For example, if you want some extra machines to do some load testing. Or if you need to store a bunch of data and want an S3-like system. Or if you want to wipe a machine and reinstall from an "image". The backup system, and backups testing. RDS, Aurora, DynamoDB, etc. Most of that can be done with a couple clicks on AWS.
For the employee cost I agree if you just have a couple boxes pumping out data your labor cost will be quite low. I'm imagining what else you might have happening on your back end using AWS services that you'd also have to replicate in your colocated installation to preserve the efficiency, which would take labor. For example, if you want some extra machines to do some load testing. Or if you need to store a bunch of data and want an S3-like system. Or if you want to wipe a machine and reinstall from an "image". The backup system, and backups testing. RDS, Aurora, DynamoDB, etc. Most of that can be done with a couple clicks on AWS.
it's $20 at DigitalOcean which is already heavily marked up :) compared to colo. The sweet spot for AWS is low traffic project that uses a very large number of AWS services or something with extremely spiky workload. It's a legend that you don't need ops people for AWS. We have a fairly complicated large scale project on AWS we have 4 full time ops people. If we were in colo we would need 5-6 instead of 4 and a bit of remote hands services.
When you buy in mbps, you're actually billed based on 95th percentile usage. So this comparison is way off, depending on traffic patterns, 1mb/s committed can work out to about 120GB in a month on average. If you use reasonable GB per mb/s numbers the cloud providers don't look all that bad.
Cannot agree more with title. I did a comparative model of AWS bills and colo bills in the context of companies of different sizes (https://blog.paxautoma.com/buy-or-rent-the-cost-model-hostin...). It turned out frequently overlooked costs for bandwidth and provisioned IOPS can be responsible of large chunk of the EC2 bill.
I suspect the high bandwidth price is a targeting tool and is primarily for repelling those wanting to host seed boxes, porn sites and the like. You can probably get a much better deal if you're paying > 10k$/mo.
Even then, cloud bandwidth is insanely expensive. For example, Hetzner offers 1.3$/TB (if you happen to exceed their generous 30 TB quota). In comparison, Amazon is 70x more expensive at 90$/TB.
Even then, cloud bandwidth is insanely expensive. For example, Hetzner offers 1.3$/TB (if you happen to exceed their generous 30 TB quota). In comparison, Amazon is 70x more expensive at 90$/TB.
Hetzner is great. I'm running a site with ~45TB monthly traffic on 3 physical servers and it costs me just over $150. I would have to pay many thousands to any cloud hosting provider and it wouldn't pay off in my case.
Sure, if high availability is critical for you business and money is not an issue, it may be worth it. YMMV.
Sure, if high availability is critical for you business and money is not an issue, it may be worth it. YMMV.
Pitchfork mode on: Outgoing bandwidth should be even more expensive! Then maybe, just maybe my mobile data cap wouldn't be drained that quickly by bloated webpages and stuff.
;)
;)
Keep in mind that with cloud providers you're also paying for the SDN that makes dynamic provisioning of VMs and logical network segmentation possible. Scalable SDN is much harder / more expensive than traditional networking.
4 to 5 times more expensive?
I'm not an expert, but it's possible... from what I understand amazon does SDN on CPUs in Dom0, Microsoft uses FPGAs, and Google has burnt their own ASICs... those are all expensive options. Much more expensive than generic networking gear. Possibly 4-5x, but idunno...
Amazon and Google's custom NICs should be no more expensive than standard NICs. Microsoft's FPGA frankenNIC is definitely expensive though.
Still, recouping the cost of the NIC by making internal bandwidth free and Internet transfers really expensive is kind of a weird way to do it.
Still, recouping the cost of the NIC by making internal bandwidth free and Internet transfers really expensive is kind of a weird way to do it.
I believe Amazon uses commodity NICs + iptables in Dom0 (based on rumors). I have no idea what a custom ASIC would cost, but I think the Jupiter networking gear / Titan NIC that Google built are pretty advanced components that would cost more than commodity stuff.
In any case, the hardware components are fixed costs. What's not fixed is the cost of applying routing and segmentation rules for each packet. There are scarce resources involved here: the hardware has limited memory and limited cycles. You're paying for the use of those scarce resources (e.g., having a memory resident stateful firewall rule for a TCP session). Charging for bandwidth isn't perfect, but it probably correlates pretty closely with the underlying resources and is a much more intuitive unit-of-value for customers.
In any case, the hardware components are fixed costs. What's not fixed is the cost of applying routing and segmentation rules for each packet. There are scarce resources involved here: the hardware has limited memory and limited cycles. You're paying for the use of those scarce resources (e.g., having a memory resident stateful firewall rule for a TCP session). Charging for bandwidth isn't perfect, but it probably correlates pretty closely with the underlying resources and is a much more intuitive unit-of-value for customers.
According to whom?
According to me... Are you arguing that it's not? It stands to reason that dynamic network architectures (SDNs) would be harder and more expensive to design and build than static architectures... it's the difference between managing shared mutable state and managing shared immutable state. On the hardware side, a simple commodity hub or switch is cheaper than a CPU, an FPGA, or burning your own ASIC to do fast SDN.
Screwing their customers? What kind of entitled attitude is this? This is a highly competitive market and the customers are voting with their own dollars. Don't like it? Don't eat it.
[deleted]
FWIW cost for a small biz at most major metro US facilities is closer to $1/mbit for a multi-carrier (which means generally multi-route and high quality with some caveats), and $0.20 if you do something like he.net. For higher volume customers you can easily cut both of those rates in half right now. And you can also participate on public peering switches for generally just a low setup fee at the best facilities.
AWS, GCE, azure seem like the platforms of yesteryear in all dimensions when compared to something like packet.net. I think these providers could be in a rock and a hard place due to the unsuitability of native Linux containers for secure multi-tenancy. This does leave a nice runway for Joyent as both a provider and software vendor for at least a little bit, but I think packet.net is really going to change the economy of infra.
AWS, GCE, azure seem like the platforms of yesteryear in all dimensions when compared to something like packet.net. I think these providers could be in a rock and a hard place due to the unsuitability of native Linux containers for secure multi-tenancy. This does leave a nice runway for Joyent as both a provider and software vendor for at least a little bit, but I think packet.net is really going to change the economy of infra.
While public bandwidth is indeed significantly more expensive on clouds, not all traffic needs to be public and different clouds charge different amounts.
I wrote a blog post on Google Cloud latency and pricing across zones and regions which may be useful for others:
https://blog.elasticbyte.net/comparing-bandwidth-prices-and-...
I wrote a blog post on Google Cloud latency and pricing across zones and regions which may be useful for others:
https://blog.elasticbyte.net/comparing-bandwidth-prices-and-...
It's suffice to say that the cloud providers have a different set of customers in mind. I have servers on both OVH/Linode variety of service providers as well as one single app running on AWS. For the products I run on OVH/Linode, I sell the service at less than $20/month. The one on AWS sells for $200+ per month. Again, it's because of the requirements/SLAs. Based on experience, AWS is a lot more robust for what I'm using it for.
As others have pointed out, this view is extremely short sighted.
To add to the mix - what if you need a multiple data-center deployment/replication? Both Amazon and Google will provide you a greatly discounted traffic there $0.02 / $0.01. And that's only start. You can easily migrate from one data center to another, with no or little cost attached to it (try that in colo).
To add to the mix - what if you need a multiple data-center deployment/replication? Both Amazon and Google will provide you a greatly discounted traffic there $0.02 / $0.01. And that's only start. You can easily migrate from one data center to another, with no or little cost attached to it (try that in colo).
I've always figured that the point of this was to allow a) overall costs to generally scale with the "size" of the customer while simultaneously creating a sticker shock effect on migrating out of whatever cloud. For example, looking at Google, the cost to transfer a terabyte out of their Cloud Storage product is six times the cost of just keeping it there for month. Of course some of this collapses if you really look into it (e.g. you are going to pay that egress anyway if people are actually accessing the data) but I'm not sure that that is always clear to execs doing back of envelope math. I think that to some degree this desire for lock-in is explicitly visible in the asymmetric ingress/egress pricing but I do think that it's a little bit underhanded if I am right because it would mean that slightly-deflated e.g. instance prices would be subsidized by lockin.
This subject is never productive on HN because almost every reply argues with a certain use case in mind but people never actually outline that use case. Those who read your post and reply do so with their own use case in mind and obviously what that other person suggested is madness (in your scenario that you never actually verbalize). Ultimately no one learns anything because they all think everyone is in exactly the same scenario as they are. Or maybe they think their personal choice is a "silver bullet". Probably depends on the poster.
Then it all repeats next week.
Then it all repeats next week.
[deleted]
I would imagine contention ratios would also factor into pricing. At least in Australia you might have a 8:1 ratio of actual bandwidth available for a consumer plan, and 3:1 for a business plan. I'd imagine that you pretty much get all the bandwidth you pay for in a data centre. I've certainly saturated 100mbps connections on servers in the past, 24x7. But perhaps people more knowledgeable than me could comment on this?
Fundamental methodological error: It compares provisioned capacity (colo & Google Fiber) to utilization.
In order for this comparison to be valid you'd need to get 100% utilization of your colo or Google Fiber pipe. You only pay for what you use with AWS et al. And quite obviously the pricing of GF and Amazon Lightsail assumes less than 100%. Nobody's getting "screwed".
In order for this comparison to be valid you'd need to get 100% utilization of your colo or Google Fiber pipe. You only pay for what you use with AWS et al. And quite obviously the pricing of GF and Amazon Lightsail assumes less than 100%. Nobody's getting "screwed".
[deleted]
I mentioned this in the price reductions of S3 announcement
People fail to realize the true cost of operating on S3, specifically when hundreds of TB of usable data is in play
"By putting the "tax" on bandwidth, a lot of these business cases are solved. I see why Amazon does that. AWS is great, but as you get into high scale (specifically in storage - 2PB+), it becomes extremely cost prohibitive."
People fail to realize the true cost of operating on S3, specifically when hundreds of TB of usable data is in play
"By putting the "tax" on bandwidth, a lot of these business cases are solved. I see why Amazon does that. AWS is great, but as you get into high scale (specifically in storage - 2PB+), it becomes extremely cost prohibitive."
[deleted]
As the author of this post I need to clarify something. I love Amazon AWS, and I love the flexibility and awesomeness of cloud computing. I just don't like the bandwidth pricing ;) Sorry for the interruption, feel free to continue crucifying me. P.S. If someone has more accurate data I'd be happy to update the post or add a guest post. Cheers, Love Arador xoxox
If colocation actually cost that much, it would make sense for a connection that allows extreme bursting to charge 3x as much per byte.
The real number to compare to is the google data for business rate. You can do lots of colocation in that price range. And that is why the cloud prices are unreasonable.
The real number to compare to is the google data for business rate. You can do lots of colocation in that price range. And that is why the cloud prices are unreasonable.
New fun:
Take credit card churning and apply it to cloud data. Build tools to seamlessly move apps between cloud providers.
Take credit card churning and apply it to cloud data. Build tools to seamlessly move apps between cloud providers.
Since Lightsail is mentioned, it's probably worth including Digital Ocean since they offer almost identical network transfer for the money [1].
[1] https://www.digitalocean.com/pricing/
[1] https://www.digitalocean.com/pricing/
It's also worth noting that Digital Ocean gives you half the amount of RAM that Linode[0] and Vultr[1] offer for the same price point.
[0] https://www.linode.com/pricing
[1] https://www.vultr.com/pricing/Amazing to see the number of people who try hard to justify this blatant rip off pricing. This is coming from the same group of people who complain endlessly about the cost of wireless data and telco data caps.
With telcos, there are only a few companies to choose from. If you want to run your own dedicated servers, nothing is stopping you. So what if people want to pay more for the ease of cloud platforms?
> Google Fiber for Business
Maybe worth renting an office in Provo just to get the deal.
Maybe worth renting an office in Provo just to get the deal.
SLC if your employees like the mountains but might want a beer or caffeinated beverage once in a while ;)
I'm not cruel enough to locate actual people in Provo, just servers.
The LDS Church has a prohibition on hot beverages but cold caffeinated are actually allowed. It is a common misconception that all caffeine is not allowed. Homebrewed beer, and homebrewed wine are technically allowed also. Many also ignore the part in the wow about eating meat sparingly while misinterpreting the rest.
As an ISP architect, I think you overlook a great many cost centers in your apples-to-doughnuts explanation.
So is Lightsail really worth jumping into?
Houses are such a rip-off! Just look how much more expensive they are than a pile of wood!
"Amazon EC2, Microsoft Azure and Google Gloud Platform are all seriously screwing their customers over when it comes to bandwidth charges."
Disagree. There's no false advertising here, they're making you pay for their service and convenience of using a combined [Paas, Iaas, Saas ..etc]. It's unfair to view these services as a singular function, you typically touch MANY features/products in production. The cost includes the convenience of offering everything under one roof, because, face it, doing everything by yourself at the SLAs provided by the giants is no trivial task.
Unless you're a BIG company that likes to distract itself with infrastructure instead of building and sharpening the core offerings, chances are that you will NEVER really build anything as reliable, inter-operable, configurable and manageable at cost.
Disagree. There's no false advertising here, they're making you pay for their service and convenience of using a combined [Paas, Iaas, Saas ..etc]. It's unfair to view these services as a singular function, you typically touch MANY features/products in production. The cost includes the convenience of offering everything under one roof, because, face it, doing everything by yourself at the SLAs provided by the giants is no trivial task.
Unless you're a BIG company that likes to distract itself with infrastructure instead of building and sharpening the core offerings, chances are that you will NEVER really build anything as reliable, inter-operable, configurable and manageable at cost.
This is true, but I think there's another point that's even more significant: not all network access is the same. In particular, putting some machines in a rack in a colo and connecting to the nearest Internet exchange is really not comparable to Google's networking, which is essentially a worldwide second Internet that is faster and more reliable than the normal Internet, and which carries your packets as far as possible towards their destination before dropping them off at a POP.
(Note: I worked for Google until 3 days ago.)
(Note: I worked for Google until 3 days ago.)
Can't agree more. I balked at the cost (still sort of do as a GCP user) until I joined Google and was able to peek behind the curtain. The network quality is absurd. GCP customers pay a premium for having the same network reach as Google. Their network works extremely hard to get user traffic onto the internal network as early as possible.
Not only that, but Google's datacenters have 24/7 attention, lots of redundant providers, etc.. A bargain basement colo won't be nearly as reliable.
Not only that, but Google's datacenters have 24/7 attention, lots of redundant providers, etc.. A bargain basement colo won't be nearly as reliable.
Yes, you're right, and you mentioned something else that I forgot to. A lot of people here are comparing Big 3 network costs to the cost of hiring a single network administrator. The correct comparison is to the cost of hiring a team big enough to staff a 24-hour on call rotation with a 5 minute response time SLA.
...and how many people need that? I would gladly 1/3 of the cost for 4 nines over 5 nines. (or whatever SLA they offer).
Not everyone need a ritzy ultra reliable network. It costs way too much and hardly any customer is going to get comparable value out of it.
Not everyone need a ritzy ultra reliable network. It costs way too much and hardly any customer is going to get comparable value out of it.
Their business customers probably really do care. Companies run off of AWS and GCP. I'm sure that spotify and Snapchat and Netflix get considerable value from those guarantees.
I agree! This is a real issue for cloud providers currently. You shouldn't have to pay for the super-reliable fast network if you don't want it.
And people don't have to! People can self host instead :-)
I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and CGI was all there was, even then uptime was so good that it was rare to hit a website that was down.
There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard to deny what they're saying and get taken seriously.
It's like six sigma, agile, or stack ranking. The big guys are doing it so it must be the right thing to do... Right? Until everyone realized it's mostly a ploy to make money selling books and conference tickets, or in this case rent out a bunch of excess capacity for huge profit.
I disagree with reliability as well. Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. With the cloud you have to worry a lot more about fault tolerance whereas with dedicated equipment simple offline backups are often enough to meet any reasonable SLA.
There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard to deny what they're saying and get taken seriously.
It's like six sigma, agile, or stack ranking. The big guys are doing it so it must be the right thing to do... Right? Until everyone realized it's mostly a ploy to make money selling books and conference tickets, or in this case rent out a bunch of excess capacity for huge profit.
I disagree with reliability as well. Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. With the cloud you have to worry a lot more about fault tolerance whereas with dedicated equipment simple offline backups are often enough to meet any reasonable SLA.
> Before all these cloud providers websites were not noticeably less reliable than they are now.
Before all of these providers existed:
1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale.
2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service)
> Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two.
3) What's your definition of "most"? Sources for those numbers please?
Before all of these providers existed:
1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale.
2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service)
> Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two.
3) What's your definition of "most"? Sources for those numbers please?
As we've gotten more device computers have gotten more powerful. Server software has gotten better. HTTP has Keepalive and multiplexing now. Encryption and networking are offloaded to hardware and we have a lot more cores.
I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets.
Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly.
I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds.
Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets.
Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly.
I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds.
Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
Thanks for the info very useful, I didn't realise how unreliable AWS is for database servers. You're correct random unannounced shutdowns of db servers is just not acceptable for critical data. I will be launching a new business based on Postgres soon and the thought of this is terrifying. I'm not keen on the RDS type services or CoLo so this is an unexpected problem I need to overcome. Do you know whether VPS provider such as Digital Ocean or CloudSigma would be more reliable?
I would say colocation is most reliable. I'm sure dedicated VPS is better but most still reserve the right to pull the plug for hardware replacements. Colo isn't terribly expensive if you buy used equipment.
Really consider how important 100% uptime is though. Google and S3 have gone down multiple times without killing the internet or losing a ton of customers. Plenty of large SaaS providers still use maintenance windows. Heck, GitHub went down today. Not sure if you use ADP but that goes down for a couple days a week!
I know it's not the popular thing to do but you can get much better relative reliability by running a single database per tenant and running a limited number of tenants per VM.
Really consider how important 100% uptime is though. Google and S3 have gone down multiple times without killing the internet or losing a ton of customers. Plenty of large SaaS providers still use maintenance windows. Heck, GitHub went down today. Not sure if you use ADP but that goes down for a couple days a week!
I know it's not the popular thing to do but you can get much better relative reliability by running a single database per tenant and running a limited number of tenants per VM.
I think you're being a bit rose-colored glasses here if you're implying that apache/cgi sites had some sort of incredible uptime record back in the time when "the slashdot effect" was still a thing.
> Most of my bare metal and colocated machines have uptime of many years.
Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!
> Most of my bare metal and colocated machines have uptime of many years.
Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!
> Nobody cares about host uptime, that just means you're not applying security updates.
Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare.
Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.
Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare.
Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.
You're comparing the simplist possible scenario with a platform that can replace a large data center.
How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs?
The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes.
Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.
How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs?
The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes.
Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.
You don't need anything but regular PC's these days...
2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap.
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years
Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right.
Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works.
Redundant transit handled by datacenter level multihoming.
Extra redundancy using dual data centers if you want with DNS round robin.
Buy dedicated lines to avoid bandwidth costs.
You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP.
Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation.
Now everybody can have a great setup as long as they pay out the nose.
Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching.
You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years
> Disagree. There's no false advertising here
I don't see any claims about false advertising. Just complaints about one aspect of the service being very overpriced. Sounds like you consider it worth it for your application, and that's great. But it is simply not priced appropriately for a lot of services, which naturally go elsewhere.
It seems obvious to me that bandwidth is Azon's proxy measure for "utility". Lots of companies do this - Oracle uses core-count as a proxy for the "utility" you get from their software, for example. But like any proxy for a different variable, it is going to be wrong; sometimes a little, sometimes a lot.
And you hardly need to be a "BIG" company to choose something other than cloud hosting. My last two looked at and rejected them. Neither qualify as even medium-sized. Both are extremely cost-aware. Running our own data centers is cost-competitive at my current gig and was much, much cheaper at the last, which was very bandwidth-heavy.
I don't see any claims about false advertising. Just complaints about one aspect of the service being very overpriced. Sounds like you consider it worth it for your application, and that's great. But it is simply not priced appropriately for a lot of services, which naturally go elsewhere.
It seems obvious to me that bandwidth is Azon's proxy measure for "utility". Lots of companies do this - Oracle uses core-count as a proxy for the "utility" you get from their software, for example. But like any proxy for a different variable, it is going to be wrong; sometimes a little, sometimes a lot.
And you hardly need to be a "BIG" company to choose something other than cloud hosting. My last two looked at and rejected them. Neither qualify as even medium-sized. Both are extremely cost-aware. Running our own data centers is cost-competitive at my current gig and was much, much cheaper at the last, which was very bandwidth-heavy.
If every cloud service arguably overprices one essential part of delivery, bandwidth usage in this case, my instinct (no more, no less and no pretension to analysis) is that there's a design intent to ensure a difficult to price component of the service provision is paid for.
I'm merely speculating, but if you have a idea what is the possible difficult to price component, can anyone help with any suggestions?
Edit to suggest a similitude: my job is to help people understand the price of (print) advertising. This is often intractable. To a certain extent, a good deal of people involved just throw costs into commission rates and other factors that make understanding the price models difficult and sometimes completely opaque. Is it impossible to imagine that the cost of cloud bandwidth is nothing less innocent than "put our non itemized expenses here'?
I'm merely speculating, but if you have a idea what is the possible difficult to price component, can anyone help with any suggestions?
Edit to suggest a similitude: my job is to help people understand the price of (print) advertising. This is often intractable. To a certain extent, a good deal of people involved just throw costs into commission rates and other factors that make understanding the price models difficult and sometimes completely opaque. Is it impossible to imagine that the cost of cloud bandwidth is nothing less innocent than "put our non itemized expenses here'?
This is simply untrue for bandwidth charges if you're pushing a lot of data.
A streaming video service, for example, might end up paying $0.10/user/hour. So a movie-a-night customer would cost $6. What multiple of reasonable do you think that is?
A streaming video service, for example, might end up paying $0.10/user/hour. So a movie-a-night customer would cost $6. What multiple of reasonable do you think that is?
About 2-3x. I'm a big fan of cloud computing in general (and AWS in particular as the clear leader, IMO) and helped lead our transition out of colos and into AWS, but the egress charges is the biggest area of AWS that I feel is egregiously over-priced.
Fortunately, it's relatively easy to locate some of your very high egress services outside of AWS. I discourage that for casual optimization, but when you get to Netflix/20 scale, that makes sense to trifle with.
Fortunately, it's relatively easy to locate some of your very high egress services outside of AWS. I discourage that for casual optimization, but when you get to Netflix/20 scale, that makes sense to trifle with.
Speaking of Netflix:
“The best way to express it is that everything you see on Netflix up until the play button is on AWS, the actual video is delivered through our CDN,” Netflix spokesperson Joris Evers said.[1]
[1] http://www.networkworld.com/article/3037428/cloud-computing/...
“The best way to express it is that everything you see on Netflix up until the play button is on AWS, the actual video is delivered through our CDN,” Netflix spokesperson Joris Evers said.[1]
[1] http://www.networkworld.com/article/3037428/cloud-computing/...
[deleted]
A good case study is Dropbox. They didn't build any of their own infrastructure until they became big enough (2 years ago).
The fact here isn't quite right. (I'm an employee at Dropbox)
Primarily, until 2 years ago we did lean on S3 for all block storage, but most of the rest of the infrastructure (metadata storage, etc) ran in our own datacenters.
Your point I think you're getting at sounds like something I'd agree with though -- you can wait a bit the cost efficiency starts to be what is important/impactful to work on before shifting your usage away from some of these providers.
Primarily, until 2 years ago we did lean on S3 for all block storage, but most of the rest of the infrastructure (metadata storage, etc) ran in our own datacenters.
Your point I think you're getting at sounds like something I'd agree with though -- you can wait a bit the cost efficiency starts to be what is important/impactful to work on before shifting your usage away from some of these providers.
I heard a story once that Dropbox started to move their data out of S3 and AWS rate limited them so they couldn't.
I don't know if it's true or not but I heard the story.
I don't know if it's true or not but I heard the story.
Dropbox built their own infrastructure over five years ago. They just took several years to turn the infra from idle to being used.
[deleted]
would a scale streaming customer really pay that much?
This is definitely true but I think that possibly as the big cloud players devote more and more of their revenue towards developing all kinds of (vendor lockin) APIs each of which does not concern the average user, rather than generic and reliable infrastructure that benefits all users, the value proposition will get worse and worse.
This comes off as a bit of a red herring. No one is denying the value of their other offerings. It's the cost of bandwidth which is clearly disproportionate.
Let them charge for extra for the 'value adds', what is the justification for adding these costs to the bandwidth specifically?
I think people raising the issue are just seeking more transparent pricing.
Let them charge for extra for the 'value adds', what is the justification for adding these costs to the bandwidth specifically?
I think people raising the issue are just seeking more transparent pricing.
Considering my experience with google adwords service, I wouldn't be surprised if the teams providing these services were under heavy pressure to over-promise, under-deliver, and sneak in as many hidden charges as they possibly can.
Did he factor in latency? if you want to ride in first class you have to pay more.
This is a super irritating feature of "enterprise" vendor pricing. What almost everyone on these platforms is doing is moving most of their bandwidth out of one of the CDN services (like say cloudfront) and then negotiating custom pricing on that bandwidth that is often as much as a full decimal place cheaper as long as you sign a couple grand a month yearly commit. There's still this massive massive pricing cusp between using the cloud as a utility and jumping into the suits & drinks & lunches sales guy game.
The big cloud platforms offer a rich selection of different offerings, which (just like in every other industry) cross subsidize each other.
When I go to a restaurant, I don't expect that they will be making the same profit margin on every item on the final bill, and in fact, they almost never do. Drinks tend to have a very high profit margin, some labour intensive items may be a break even at best, and the complimentary bread sticks or chips and salsa (if offered) will certainly be a loss.
I guess I could write a very upset article about how my local mexican restaurant is SERIOUSLY SCREWING ME OVER with their drink prices, but if I don't write the companion piece about their cheap burritos (subsidized, of course, by the drink prices), it would only show half the picture.
The reality is that I'm buying a whole package (at AWS or a restaurant) and I should evaluate the whole picture. Yes, I can get bandwidth cheaper outside AWS (or a can of coke a lot cheaper from a big box retailer). But I can't really get the total package of integrated, managed services outside AWS (certainly not for the cost they charge), any more than I can get someone else to show up in my kitchen and cook a three piece meal and then do all the dishes. (Which is to say, I totally could hire a chef to do that, but it would cost me a lot more. I could BUILD an internal SQS clone if I had to, but my employer would never break even on the cost of getting me to do so.)
AWS is very cheap for some things and very expensive for others. Depending on your usage and workload it may or may not be economical to buy the package they offer. If it is, go for it. If not, don't. Just like, you know, every other good or service you purchase in both your personal and professional life.