Netflix, Long an AWS Customer, Tests Waters on Google Cloud(theinformation.com)
theinformation.com
Netflix, Long an AWS Customer, Tests Waters on Google Cloud
https://www.theinformation.com/articles/netflix-long-an-aws-customer-tests-waters-on-google-cloud
145 comments
You don't have to be Netflix to get AWS, GCP and MS to play against each on pricing. I've seen a number of my clients build a prototype on each platforms, and then use it to make final decision. Part of that final decision is having the cloud vendors compete on price.
I personally haven't seen any massive price breaks, but there definitely is room the margins.
I personally haven't seen any massive price breaks, but there definitely is room the margins.
For years Dell would let it "leak" out to the press that they were looking at supporting AMD chips. It would happen every year like clockwork. Then after a month you'd hear that Dell and Intel had come up with a new deal to remain Intel exclusive.
Hasn't Dell offered AMD chips for a long time? Dell currently sells AMD-based laptops, and was doing so back in 2008 as well.
For example, this Ryzen laptop: http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-...
For example, this Ryzen laptop: http://www.dell.com/en-us/shop/dell-laptops/new-inspiron-15-...
Before then – in the 90s both Intel and Microsoft used a lot of backroom deals to keep big vendors exclusive. If memory serves it took the Opteron’s crushing performance advantage before we started seeing that break for Intel servers; Linux users for years were stuck paying for OEM Windows installs which were never used.
Yes, but I've been around the industry a while. I specifically remember this happening during the late 90s and just in to the 2000s.
It was occurring at a time when AMD was fairly firmly beating Intel on a straight $/performance basis (~K6 era). If Dell had started shipping AMD based systems back then it could have been a big game changer.
The general narrative around the tech press at the time was that Intel knew this and were desperate to keep Dell as Intel exclusive.
It was occurring at a time when AMD was fairly firmly beating Intel on a straight $/performance basis (~K6 era). If Dell had started shipping AMD based systems back then it could have been a big game changer.
The general narrative around the tech press at the time was that Intel knew this and were desperate to keep Dell as Intel exclusive.
These are probably to prevent action from the FTC.
I think they are talking about when Dell made serious profits in PCs.
Really playing these companies against each other would mean dynamically balancing your services across the providers based on pricing
I work for a large CDN. Lots of big customers do this, they shift their traffic around between different CDNs based on a lot of metrics, including price and performance.
I also work for a CDN.
Some of our stuff dynamically utilizes other CDNs. A customer might go try and test deploy something and we might use (AWS or whatever) to do it.
It's CDNs all the way down.
Some of our stuff dynamically utilizes other CDNs. A customer might go try and test deploy something and we might use (AWS or whatever) to do it.
It's CDNs all the way down.
I work for a company people use to build CDNs. Just another turtle. :D
like physical infrastructure or something else? a bit curious (having many neteng friends at many, many CDNs)
I have similar friends!
We the whole stack for most people, but also run our software on "private cdns" for some. We primarily target devs: https://fly.io
We the whole stack for most people, but also run our software on "private cdns" for some. We primarily target devs: https://fly.io
Adding my turtle as a CDN employee ;-)
We license software[1] to ISPs/MSPs who then use it to "OEM" Kubernetes clusters to CDN providers.
We should rename our product "Infinite Regress" ;-)
[1] https://gravitational.com/telekube/
We should rename our product "Infinite Regress" ;-)
[1] https://gravitational.com/telekube/
I wonder how many of the people in this chain are sitting next to me.....
I've love to see some open source tools that manage this automagically for me. :-)
It would also require an architecture that supports multiple interfaces. Totally doable (the facade pattern comes to mind) but you'd want to calculate the TCO to see it works out financially.
It would also require an architecture that supports multiple interfaces. Totally doable (the facade pattern comes to mind) but you'd want to calculate the TCO to see it works out financially.
Shameless plug: we have an open source tool you can build stuff like this on. It'll run any logic you can implement in JavaScript, we have people doing things like "send traffic to backend with lowest latency" already, it'd be pretty easy to pull in some API data to pick based on cost.
https://github.com/superfly/fly
https://github.com/superfly/fly
[deleted]
Did this a lot with payment processors. Literally had application router settings that could route based on % or switch off once a certain monthly or yearly amount was hit. Configs could be updated in real time.
If it's a final decision, then most of the cost is in the future, far beyond the reach of the initial quotes.
I've seen too many companies turn extractive (including, of course, that same Microsoft) to ever use the words "final" next to "platform decision".
I've seen too many companies turn extractive (including, of course, that same Microsoft) to ever use the words "final" next to "platform decision".
I agree with most of what you said, however, Netflix most definitely does not operate at a scale well beyond what most AWS managed services can address.
AWS managed services literally sit behind billions of devices, a scale that makes Netflix look small in comparison: https://techcrunch.com/2018/02/27/apple-now-relies-on-google...
AWS managed services literally sit behind billions of devices, a scale that makes Netflix look small in comparison: https://techcrunch.com/2018/02/27/apple-now-relies-on-google...
Seriously? or are you a member of AWFUL - Americans Who Figuratively Use Literally ;)
Literally has been used as an intensifier for a figurative expression since the 1800s and has been defined in the dictionary that way for quite a while.
It was used in this manner by Charles Dickens, James Joyce and American authors as well.
Although, it pisses me off when I hear it used that way.
It was used in this manner by Charles Dickens, James Joyce and American authors as well.
Although, it pisses me off when I hear it used that way.
> (on which they already get a huge discount)
just wondering, how huge (as a percentage) that discount is?
just wondering, how huge (as a percentage) that discount is?
They'll play a few games with contracts to spread the costs around (e.g. giving n% discounts for m years of reserved capacity). You probably won't get much just by saying "I can just go to GCP" and asking for the world. The more you spend, the more distinct services you buy, and the longer you want to commit the steeper the discount. I've seen up to 10% but I'm sure they do more for bigger customers.
I've seen upwards of 50% for some particularly large customers. To the point of "if we lose our discount, our company goes under".
Makes you wonder if the vendor is still making money on the deal.
Margins are pretty high, in part due to scale. If AWS were to hypothetically sell 80% of capacity at cost, that makes the last 20% more profitable than it would be to just sell the 20% alone.
Reserved instances always make money because you make profit without depreciation kicking in.
They get CAF. When CAF runs out, pricing shoots up.
What's CAF? Googling wasn't very helpful :)
Customer Appreciation Funds/Customer Acquisition Funds/etc.
"We have $500M available this year that we can spend to lock in $1B in revenue. If you give us $1B in revenue, we would spend $500M on helping you to achieve your objectives. If you give us $100M in revenue, we give spend $50M in helping you to achieve your objectives"
So say I need to spend $10M on some project and I am looking at $20M of spend on hosting. You are offering me 10M CAF. I spend $20M on hosting, and 0 on some other project - instead you spend your 10M CAF on helping me with my project, which of course looks really good on my books.
"We have $500M available this year that we can spend to lock in $1B in revenue. If you give us $1B in revenue, we would spend $500M on helping you to achieve your objectives. If you give us $100M in revenue, we give spend $50M in helping you to achieve your objectives"
So say I need to spend $10M on some project and I am looking at $20M of spend on hosting. You are offering me 10M CAF. I spend $20M on hosting, and 0 on some other project - instead you spend your 10M CAF on helping me with my project, which of course looks really good on my books.
Firstly: not very new. Hasn't Netflix been at this for a long time?
Secondly, everyone should be checking. Not only is Google doing a lot of subsidization, but their services are really really good. My current employer ended up being unable to use them but it's I acutely regret every day I'm forced to build out things Google already has or deal with Amazon's frustratingly antiquated ELB infrastructure.
Secondly, everyone should be checking. Not only is Google doing a lot of subsidization, but their services are really really good. My current employer ended up being unable to use them but it's I acutely regret every day I'm forced to build out things Google already has or deal with Amazon's frustratingly antiquated ELB infrastructure.
AWS's ELB/ALB have been rock solid for us. Not sure what you are referring to when you say 'antiquated infrastructure'?
To answer your question directly, AWS's poor spike handling has really made me distrustful of them. It's not that AWS's support staff is bad (actually, they're phenomenal given the difficulty of that job) but... the fact that I can be taken down by a big scale event because I can't pre-scale my ELBs without support help that takes hours is kinda annoying.
Now, I've been TOLD by ___________________________________ this is actually because it's a pain in the butt and they're working to fix it, but AWS's budget seems to favor new shinies over iterating on the existing environment. Look how long it took us to get a new and less buggy AWS dashboard and that still hasn't propagated to all the internal services.
I could go on. For example, my experience is that GCE's and even Azure's I/O and networking is way faster, more consistent and more reliable than AWSs. I host all my dev boxes on GCE because using an attached drive (so I can de/re-provision to save on costs) is much less agonizing. Also: google has some AMAZING database products that category kill whole infrastructure tasks. It's a very good kind of vendor lock-in when they can say, "You can't move away because no one else has found a way to provide our feature set."
Now, I've been TOLD by ___________________________________ this is actually because it's a pain in the butt and they're working to fix it, but AWS's budget seems to favor new shinies over iterating on the existing environment. Look how long it took us to get a new and less buggy AWS dashboard and that still hasn't propagated to all the internal services.
I could go on. For example, my experience is that GCE's and even Azure's I/O and networking is way faster, more consistent and more reliable than AWSs. I host all my dev boxes on GCE because using an attached drive (so I can de/re-provision to save on costs) is much less agonizing. Also: google has some AMAZING database products that category kill whole infrastructure tasks. It's a very good kind of vendor lock-in when they can say, "You can't move away because no one else has found a way to provide our feature set."
Antiquated doesn't mean unstable or unusable. It means old fashioned. It's not the new hotness.
I see such intense benefits from HTTP/2 end to end, I'm quite happy to be called a shiny-chaser when my 50th and 99th drops so sharply.
Also, prefetch is absolutely incredible for webapps and I'm trying to get my peers at my current employer to embrace it even if it's not well supported in AWS.
Also, prefetch is absolutely incredible for webapps and I'm trying to get my peers at my current employer to embrace it even if it's not well supported in AWS.
Have you tried termination at ELB with TCP to backend, or just plain TCP passthrough all the way?
Also as mentioned elsewhere, ALB or NLB might do what you want, either working the same way as regular ELB or via the same workarounds.
Also as mentioned elsewhere, ALB or NLB might do what you want, either working the same way as regular ELB or via the same workarounds.
I’m sure you are aware but AWS has a newish loaf balancer now: https://docs.aws.amazon.com/elasticloadbalancing/latest/appl...
What are the problems you've been running into with ELB?
No http2 support is increasingly limiting
ALB has HTTP2 support.
Only at the front end, it downgraded to 1.1 between the ALB and your servers.
All load balancing services do that today. It is rare to get HTTP/2 on both ends unless you run your own software like Envoy.
There are a lot of comments saying that Netflix should run its own datacenter.
Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS
https://www.linkedin.com/in/adriancockcroft/
He did an interview with Software Engineering Radio back in 2014 explaining their motivation.
http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...
Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS
https://www.linkedin.com/in/adriancockcroft/
He did an interview with Software Engineering Radio back in 2014 explaining their motivation.
http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...
> Adrian Cockcraft helped lead Netflix's effort to move to AWS and now is a VP at AWS.
I always get a bad taste in my mouth when I read things like that. Was it really an objective choice, or was this AWS’ way of saying “thank you” ?
I always get a bad taste in my mouth when I read things like that. Was it really an objective choice, or was this AWS’ way of saying “thank you” ?
You're ignoring the fact that he spent a few years at Battery Ventures in between Netflix and AWS. Adrian is a well respected professional in his field, has been around for decades, and nobody holds any negative perceptions of him.
He wasn't hired by AWS because he migrated datacenters to the cloud. He was hired by AWS because he created this: https://netflix.github.io/ - Netflix's OSS, which has allowed them to attract so much great talent, was a direct result of Adrian's passionate open source advocacy. He's doing the same thing at AWS now, and has already accomplished great things like open sourcing all of the AWS documentation.
Disclaimer: work at AWS.
He wasn't hired by AWS because he migrated datacenters to the cloud. He was hired by AWS because he created this: https://netflix.github.io/ - Netflix's OSS, which has allowed them to attract so much great talent, was a direct result of Adrian's passionate open source advocacy. He's doing the same thing at AWS now, and has already accomplished great things like open sourcing all of the AWS documentation.
Disclaimer: work at AWS.
No way, if I was CTO of a big company (or, say, whoever at Pentagon is in charge of IT), I would love to speak to the guy that shifted Netflix away from in-house datacenters to "The Cloud".
He led the largest migration from on premises to AWS. Who else would have more experience?
And now dropbox owns those datacentres. ;)
A lot of this article isn't that surprising -- especially the part where they're keeping "Business Critical" data on GCP. Of course they are! Every company needs a disaster recovery plan and in general shouldn't put their eggs in one basket.
I'd be interested to see where the AI/ML frameworks differ though, and what additional value one can extract using GCP versus AWS and at what cost/presumptions.
I'd be interested to see where the AI/ML frameworks differ though, and what additional value one can extract using GCP versus AWS and at what cost/presumptions.
My prediction is that by 2020 Netflix will move a significant portion of their IT to Google Cloud.
Why? Most of their architecture is microservices / linux containers / serverless, and Google is simply way better positioned there (with Kubernetes as their crown jewel). They will eventually win.
Source: no source, just my "expert" opinion (6 years at AWS, 2 at VMware's cloud, currently CEO at a startup).
Why? Most of their architecture is microservices / linux containers / serverless, and Google is simply way better positioned there (with Kubernetes as their crown jewel). They will eventually win.
Source: no source, just my "expert" opinion (6 years at AWS, 2 at VMware's cloud, currently CEO at a startup).
Another reason is the rumor that Google is going to make their own chips for their Cloud, possibly ARM based. This could have big cost savings with some applications. Video streaming and encoding are one of them.
Isn't Netflix invested heavily on Mesos than k8s for their container orchestration? for example Titus https://medium.com/netflix-techblog/tagged/titus Fenzo https://github.com/Netflix/Fenzo
While that might be the case, I wouldn’t be surprised if they let teams play with k8s as well. Besides, one should always look at these choices objectively; if Google + k8s is a better deal overall than AWS & Mesos, then you shouldnt let your earlier investments cloud your judgement.
I know both Kubernetes and Mesos quite well. To be frank, they're on two different levels - and I assume the smart(er than me) guys at Netflix know that as well.
Kubernetes is a huge deal here. The big move to containerization makes GCP very attractive to new infrastructure.
How big doesn’t netflix have to get before it’s in their best interests to build data centers of their own?
Toolchains are maturing. They’ve figured out ways to get multiple uses from the same hardware. They shouldn’t be beholden to the rollout or geographical plans of two cloud providers.
I don’t even mean build out their entire infrastructure. I think even running your base load on your own hardware makes you understand your cloud situation much more profoundly.
Toolchains are maturing. They’ve figured out ways to get multiple uses from the same hardware. They shouldn’t be beholden to the rollout or geographical plans of two cloud providers.
I don’t even mean build out their entire infrastructure. I think even running your base load on your own hardware makes you understand your cloud situation much more profoundly.
They _had_ their own datacenters, and decided to move their control plane to the cloud (AWS) to avoid having to run the those datacenters themselves.
https://media.netflix.com/en/company-blog/completing-the-net...
This was a big deal in 2008 :)
(Maybe there's an even bigger size where it no longer makes sense for them to outsource this, but bidding GCP and AWS against each other seems like a reasonable strategy for their use-case).
https://media.netflix.com/en/company-blog/completing-the-net...
This was a big deal in 2008 :)
(Maybe there's an even bigger size where it no longer makes sense for them to outsource this, but bidding GCP and AWS against each other seems like a reasonable strategy for their use-case).
They have data centers of their own and distribute all of their own content. AWS powers the UI requests and their back office number crunching stuff. Everything after you push play is done via Netflix's own hardware.
Any links so I can read more on the subject? I’d love to get my hands on some material describing their infrastructure.
That seems to be more about their edge caches, which is interesting [and partly answers one of bad assumptions] but not the same.
AWS is catching up as far as Kubernetes support. I might agree that GCP is better positioned for some of that architecture, but they are still comparable. There are also some advantages to AWS. For example, I think that their RDS product is still more mature than Google Cloud SQL.
It's all trade-offs.
It's all trade-offs.
+1 regarding RDS. A company I spoke to is moving from GCP to AWS, with their biggest pain point being Cloud SQL is WAY less flexible than RDS. While GCP's k8s offering is better/easier to use, they still have some catching up to do.
To drive the second point home, GCP recently released VPC Flow Logs, while AWS has had it for a good while now.
To drive the second point home, GCP recently released VPC Flow Logs, while AWS has had it for a good while now.
I think long term, Netflix needs redundancy, especially when the primary supplier is a competitor in the same space.
Perhaps long term, that would be an issue. But short term and even medium term, I don't see Netflix, Prime Video and YouTube Red as competition. Their real competition is cable/satellite and as long as Netflix helps round out a cord-cutting bundle that convinces people to ditch cable, it's a net benefit to Google/Amazon. And as cable dies, especially given the recent FCC direction, you can expect them to start to play more games with the internet connections they also provide to customers. I think Google and Amazon realize that Netflix will be much more sympathetic in that fight than either of them will be. So it makes sense for Amazon/Google to support Netflix as much as possible so they can use them as a proxy in the upcoming fight against the cable companies.
Honest question - is Amazon really seen as a competitor here? Every time I've attempted to use Prime video, I'd scurried away terrified of their awful ui. I guess it's hard to gauge subscribers since it's 'thrown in', but I'd be curious to know how many people really use the service.
I use Amazon Video constantly now that Netflix is mostly/hugely populated with Netflix Originals only.
Not even that, but because Amazon has a history of not playing nice!
Google also may have a competing service if they ever figure out what they want YouTube Red to be.
If they ever figure out what they want any of their products to be...
Cobra Kai is a pretty good example of the future of YouTube Red: actual Hollywood talent and directors, instead of YouTube stars with higher production values.
I'm as much a fan of Karate Kid as the next guy, but calling both Johnny and Daniel-san 'hollywood talent' is questionable. I'd not seen either of them in anything from Hollywood since the 1989 Karate Kid part 3
Yeah. It might only be a matter of time until, to avoid "customer confusion", Amazon decides to no longer have Amazon services provide things to Netflix customers?
No one would dare try to sell a set top box in the US that didn't support Netflix.
And the standing problem now is that their secondary supplier they're considering is also a competitor in the same space. It's a wonder Netflix isn't testing the waters against Azure, since Microsoft just closed their video sales offerings down.
Microsoft pulled out of the music space, they're still selling & renting movies/shows and content services are a core feature of azure. Netflix uses its own POPS for of its content delivery.
Ah yes, I mixed that up. :/ Though who rents movies from Microsoft? And they don't have a subscription-based service for it, like both Amazon and Google do.
I rent movies and tv shows from Microsoft all the time. They have some great sales.
Xbox owners.
Isn't it better to be in bed with your competitor so as to have a solid claim against being declared a monopoly?
Depending on the details, that could be considered a trust and then you'd be in the same boat as if you were declared a monopoly.
Couldn’t YouTube also be seen as a competitor?
Original YouTube content is really not a huge part of YouTube's offerings. Even if it were a larger component, Google wouldn't jeopardize a large hosting account just to score points against a competitor.
Amazon, on the other hand, will partner and make nice with competitors until they're ready to abandon them. Amazon is planning on supplanting FedEx and UPS and already has its own fleet of cargo planes. Amazon is great for business until they decide they don't need you anymore.
Amazon, on the other hand, will partner and make nice with competitors until they're ready to abandon them. Amazon is planning on supplanting FedEx and UPS and already has its own fleet of cargo planes. Amazon is great for business until they decide they don't need you anymore.
I just got back from NAB and one thing clear was that Google is aggressively trying to get into the broadcast video space with their Cloud Offering.
I would think if a business use more than 500 servers, it would make sense to start considering having your own datacenter?
It’s not either/or. You want to be in a position where your own racks and public cloud providers are all running instances of the same distributed scheduler. Then you can buy capacity wherever it makes the most economic sense at the moment. Just proving that you have the capability to run your own hardware cost-effectively can negotiate Amazon/Google down to that price or lower.
This is probably the nicest answer. With things like kubernetes, hardware is quite abstracted. If it makes sense to use your own hardware, your software should be good enough to effortlessly shift some load to your own hardware and back incase own hardware can't handle a spike. Hybrids are probably the future where you rent when it makes sense, and buy when it makes sense and move compute and storage by dragging a slider.
I wouldn't do my own datacenter as in build out a building, but I would certainly lease a cage and build out my own infrastructure at probably half that count.
As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means.
For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.
As far as I know, Netflix has POP's for content delivery in many carrier facilities so it's not a pure cloud player by any means.
For the cost of 1 year for a single cloud server you can buy a 52 core dell with 256gb ram and a few SSD's and have price/performance ratios that heavily scale in your favor and with a scheduler like marathon, k8s, nomad you can easily achieve elasticity in a private colo to a large degree without also marrying yourself to cloud specifics and nothing would stop you from peering with major cloud providers for seasonal elastic needs.
I think this is a crucial distinction. Netflix has a relatively unique architecture for their content delivery system compared to other online video providers, which is worth considering in context of this article.
With Netflix's Open Connect (https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances.
Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc."
At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.
With Netflix's Open Connect (https://openconnect.netflix.com/en/) system, most residential Internet Service Providers have appliances at the regional point of interconnection. These appliances are inside of or directly connected to the ISP's network in the region. Back in 2016 they had appliances deployed to over 1,000 locations. Content is pre-positioned during off-peak hours to the appliances.
Previously Netflix has been open about how they use AWS for "generic, scalable computing" including "all of the logic of the application interface, the content discovery and selection experience, recommendation algorithms, transcoding, etc."
At the time, they claimed that one of the advantages of doing this was the ease of use and growing commoditization of the “cloud” market, so it makes sense that they are evaluating alternative cloud providers for these commodity services.
I've used most of Hashicorp's tooling - Consul, Nomad, Vault, etc. - and I consider myself to be a Hashicorp fanboy. But if you're spinning up a bunch of VMs "in the cloud" and that's all you're doing, you're not taking advantage of the platform. The idea behind using AWS is not just to avoid your own capital infrastructure, it's also to allow someone else to do the "undifferentiated heavy lifting". In our case:
Consul -> ParameterStore
Vault -> KMS/Cognito/IAM
Nomad -> Elastic Container Services/Fargate (serverless Docker)/Lambda
SQL Server -> RDS -> Aurora
Memcached -> ElastiCache (Memcached protocol)
Consul+Fabio (service discovery/load balancing) -> Elastic Load Balancers/Route 53/Autoscaling
VSTS (MS's hosted CI/CD platform) -> AWS Code Build/Code Deploy/Code Pipeline
"not marrying yourself to a one cloud provider" is equivalent to "coding using a repository pattern to access the database" in both cases people claim to not wanting to tie themselves to one vendor and then they realize that they aren't taking advantage of what the vendor has to offer.
In my experience, no one wants to go through the pain of moving to another vendor just to save a little money.
Consul -> ParameterStore
Vault -> KMS/Cognito/IAM
Nomad -> Elastic Container Services/Fargate (serverless Docker)/Lambda
SQL Server -> RDS -> Aurora
Memcached -> ElastiCache (Memcached protocol)
Consul+Fabio (service discovery/load balancing) -> Elastic Load Balancers/Route 53/Autoscaling
VSTS (MS's hosted CI/CD platform) -> AWS Code Build/Code Deploy/Code Pipeline
"not marrying yourself to a one cloud provider" is equivalent to "coding using a repository pattern to access the database" in both cases people claim to not wanting to tie themselves to one vendor and then they realize that they aren't taking advantage of what the vendor has to offer.
In my experience, no one wants to go through the pain of moving to another vendor just to save a little money.
That's a good distinction. CoLo providers are pretty reasonably priced and most of the larger ones are good at what they do.
There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
There's a HUGE scale difference between a CoLo and your own -- as soon as you start having to negotiate diesel supply contracts yourself things get crazy fast.
I would never roll my own unless I was of facebook/google size. Colo space is abundant and affordable.
Netflix doesn't pay retail for AWS or G Cloud
This isn’t such an obvious decision. Running your own datacenter is difficult. Besides racking and stacking, you need to deal with power and cooling. Most likely you will outsource all of this to a collocations management services initially, which becomes your overhead just like AWS/Google. But they have economies of scale, in addition, they have expertise in managing all of this equipment with very high reliability. They also have tons of spare capacity (elastic) that are at your disposal, you’ll lose this in your own datacenter.
On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms.
At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen.
Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).
On top of that you need to hire specialists, and 24/7 noc teams, for making sure your site is always up. And it’s not just one, you need a minimum of two, geographically separated so that they are not effected by the same fire, earthquake, power outage from ice/wind/electrical storms.
At the end of the day, You need to beat AWS’ profit margin, and you still would have a huge amount of difficulty being able to match the uptime that you get when you’re so easily able to distribute services across regions and availability zones that are resilient to all the real world problems listed above. And all of those do happen.
Even for the largest of companies, I’m not sure it makes sense to compete with these services except in the most niche of markets (like dropbox and storage).
The cloud doesn't really have economies of scale unless your workloads scale opposite of pricing - which they very rarely do. For small shops, sure, but for the example of 500 instances running - you could lease space and grow with physical servers at a better economic scale. If you only need 500 servers for 1 busy season and can drop back down to 100 then stay in the cloud. If you need 500 always running instances and your growth is 20% year over year, i can almost guarantee a good colo will be cheaper and faster if you use the same automation and configuration as code philosophies in private dc as you would in public cloud. Puppet/foreman/cobbler/ansible/chef all work great in private DC
In general, I think it’s a wash. The margins you save on end up costing you in other ways, such as development velocity and having access to easily deployed high-scale solutions like dynamo and managed instances like RDS. it takes time and organizational know-how to get to a point where similar things can be managed in the same way as AWS.
Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.
Again, there are niche segments where this might not be true, 500 constant machines churning through massive datasets all day-long, etc, might fall into that category, but it’s not slam dunk.
The margins you're paying for include functionality you don't use and your own cost of integration, observation, and management of their stuff. The principal advantage of using a cloud provider is avoiding capital costs and any useful apis they have that you'd otherwise have to develop/maintain in-house.
The most fun thing about startup world is that it pretends that normal rules of economics and accounting do not apply to it (until a startup runs out of money)
$X in CapEx makes balance sheet look better than $X in OpEx.
$X in CapEx makes balance sheet look better than $X in OpEx.
As generously as I can put it: It's a form of denial that people use when overwhelmed with everything else. There's typically too little real data on what the money's going to look like until a bunch of things are tried out, each essentially a random variable. So instead of freaking out about that, just bandage over it with some lies to yourself until you can actually start addressing it.
Businesses in well-known markets can start estimating their costs + potential revenue early, and optimize early. But for a business that might "pivot" a few times before finding "fit", how do you do it? You just look at time (runway: burn rate * remaining capital) you have to try stuff.
It's not great, but I can understand why some people do it.
Businesses in well-known markets can start estimating their costs + potential revenue early, and optimize early. But for a business that might "pivot" a few times before finding "fit", how do you do it? You just look at time (runway: burn rate * remaining capital) you have to try stuff.
It's not great, but I can understand why some people do it.
If you need to pivot before you are cashflow positive, then you dont have a business and you never had a business.
I keep being amazed that total business failures that result in pivots are considered to be successes in the startup world.
I keep being amazed that total business failures that result in pivots are considered to be successes in the startup world.
There are lots of open source products that offer programmatic/desired state/idempotent config & apis for provisioning and hardware lifecycle management in private dc.
There are not even close to how good and mature the AWS tools are.
You really have to factor in the labor costs of infrastructure and operations into that as well. Running your own private "cloud" isn't cheap. And no matter what you do, you aren't going to get even close to feature parity with the public cloud offerings (or even speed, if you are using virtual-machines). There are a lot of trade offs that you have to weigh in that don't always come down to bare dollars and cents.
Everything you'd have to pay for, Amazon or Google has to pay for for you... plus profit margin. At Netflix scale, it's almost impossible for someone else's cloud to be cheaper than running your own, and hiring the necessary staff and acquiring the necessary space and hardware to do so.
I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
I honestly have to wonder if the level of Amazon's discount for Netflix is absurd to the tune of "we lose money doing this" just to retain them. AWS may need Netflix, to make serving it's smaller clients economically viable, more than Netflix needs AWS.
If you're a margin business then certainly. If you're not then there's a lot to be said for keeping your core competency lean and offloading the heavy lifting at a negotiated price.
This is assuming you can do everything at the same efficiency level as Amazon or Google. It's likely that you wouldn't be able to outperform on these metrics to a degree that it would make the profit margin difference a wash.
The exact point I am making is that at a certain point, you are able to do things as efficiently as they are. Netflix certainly qualifies at that scale.
It's different for everyone.
If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.
If your 500 servers are for storage, S3/GCS will beat you in cost. For compute, that might be a different story, but then you still have to consider regionality, networking, etc.
Let's be honest, even in the public cloud there are very few orgs that can survive regional failures and most people solve locality through CDN's for content delivery which means a private datacenter isn't really that different from public cloud for most orgs. Also, when when orgs move to the cloud they don't fire their networking team, they still have deskopts/laptops/servers/security devices/printers and large lans/wans to connect / peer already and these teams could staff up for private datacenter without really adding additional costs that aren't already incurred as part of doing business anyway. Our private datacenter is a few thousand servers leased space in a colo and we have one "datacenter dude" who handles hardware issues and our network team spends 90% of their time still on corporate infra vs actual supports with colo space. One can take some up front engineering/planning costs to scope out a private datacenter for about the same cost as it is to scope/plan VPC's and peering/connectivity/vpn services.
I completely agree. I jokingly tell everyone I know using AWS to "not use us-east1" and that they need to use another region not just AZs...and they always laugh and say "you were right" with us-east goes down.
At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
At least from my view, there's a difference in backbone versus corporate space networking, but you're right, most everyone won't ever realize any savings from that side of things.
How often does us-east-1 go down? I know of about 4 big outages in the last 7 years but you're making it sound like it happens much more frequently. Is there a list of outages somewhere?
it's not just about it going down.. if you rely only on us-east they have had multiple times when you couldn't autoscale because there was no more capacity... if you rely on running lean and scale/up/scale down events us-east would have bitten your ass a lot the past year... Just for comparison sake even our private colo has had 0 issues the past decade that caused any interruptions to service
I do run such a service, one that must scale daily in response to traffic at multiples of off peak traffic and I can state unequivocally that this is false.
I can vaguely recall over the past several years single availability zones having short periods of time where single instance types were unavailable, but never the entire region... The idea that this happens "a lot in the past year" for the entire region isn't my experience at all.
I can vaguely recall over the past several years single availability zones having short periods of time where single instance types were unavailable, but never the entire region... The idea that this happens "a lot in the past year" for the entire region isn't my experience at all.
It's a good question, but depends on your workload.
Netflix is all about content delivery -- which is both distributed and highly variable over the day/year.
Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.
Netflix is all about content delivery -- which is both distributed and highly variable over the day/year.
Using Cloud means Netflix can deploy closer to the customer. Plus scale up and down as required. If they did this themselves it would be quite expensive and involve a lot of redundancy.
I'm surprised they have thought to move some of their infrastructure in house. The clearly have the scale where this would make sense, but it may be that AWS and GCP provide them with such low special prices that it doesn't make sense...
Their CDN is in-house [1]. They use cloud services for general computing.
[1] https://media.netflix.com/en/company-blog/how-netflix-works-...
[1] https://media.netflix.com/en/company-blog/how-netflix-works-...
The problem might just be that it's genuinely incredibly difficult to get infrastructure right at this level, and the cost to move in-house is still too high, even accounting for the savings over the long term.
This is natural commodification of compute, especially with such large deployments where there will be a lot of in-house tooling and apps that just need raw compute and storage to run, no matter where it comes from.
The rise of containers and Kubernetes makes this even easier now. It'll be interesting to see how the clouds differentiate beyond this as most managed services are still rather underwhelming compared to the hype and are also often beat by Kubernetes deployments if available.
The rise of containers and Kubernetes makes this even easier now. It'll be interesting to see how the clouds differentiate beyond this as most managed services are still rather underwhelming compared to the hype and are also often beat by Kubernetes deployments if available.
Looks like this is paywalled.
Edit: Since this is being downvoted, here is the proof: https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinfo...
I have no issue with paywalled-content, it should just be noted that this is, at least for some readers.
Edit: Since this is being downvoted, here is the proof: https://screenshots.firefox.com/bgD6FxWvQS4e0sB2/www.theinfo...
I have no issue with paywalled-content, it should just be noted that this is, at least for some readers.
I agree. If putting [video] on video links is a thing here, so should [paywall] or similar since it's even more annoying than videos.
However, I'm a subscriber to The Information so here's a share link: https://go.theinformation.com/f2b133baedc57a8c - I think it requres an email, alas, but it's something if people are desperate to read the story.
However, I'm a subscriber to The Information so here's a share link: https://go.theinformation.com/f2b133baedc57a8c - I think it requres an email, alas, but it's something if people are desperate to read the story.
lol, This a "press release" written by a PR firm employed by google
etflix, one of Amazon Web Services’ biggest customers, is expanding its use of Google Cloud, AWS’s biggest rival, according to two people with knowledge of the matter. The move, which hasn’t previously been reported, could portend a shift in the balance of power in the cloud computing market
etflix, one of Amazon Web Services’ biggest customers, is expanding its use of Google Cloud, AWS’s biggest rival, according to two people with knowledge of the matter. The move, which hasn’t previously been reported, could portend a shift in the balance of power in the cloud computing market
This could also be titled "Netflix Ensures its AWS Discounts Continue"
AWS sales teams are quite responsive to threats of moving in terms of continuing discounts. I've drawn up migration plans solely to use as leverage against AWS for deep discounts in the past.
AWS sales teams are quite responsive to threats of moving in terms of continuing discounts. I've drawn up migration plans solely to use as leverage against AWS for deep discounts in the past.
Just curious: Can you state a range of possible discounts against list price? The stuff I deal with is small fry but I always wonder how much cheaper the big guys get their cloud services compared to us.
Third hand knowledge - since I was not a part of the negotiations - fair warning. The company I worked for was getting discounts in the 40-50% range. It was the only way they could afford to use AWS, frankly, since it was all static provisioning (i.e. no elasticity in the workload). They were also approaching a petabyte of data stored in S3.
This is one reason you don't want to go with specialized databases or queues. If you used a managed Queue or Database, make sure you can get that same exact Queue or Database on another cloud provider.
If more people start using GCP, then in the near future, GCP's prices will go up. In contrast, AWS prices may come down to compete with GCP. And when they reach close parity, things will begin to commoditize and get even cheaper across the board. I see this being a good thing overall as an industry.
Why would GCP prices go up?
We might have bigger problems if AWS goes down
Really surprised this has not happen earlier.
This is off topic, but the mention of my data going onto Google cloud made me wonder: does Netflix share viewing histories?
I read their privacy policy but it was so mealy mouthed it's hard to make heads or tails of to a non-lawyer.
I read their privacy policy but it was so mealy mouthed it's hard to make heads or tails of to a non-lawyer.
Disclosure: I work at Google Cloud on Kubeflow
I can't speak for Netflix's privacy policy, but, if your concern is with Google, we never ever ever ever mix/merge Google customer data with customer data. Like ever ever ever.
FWIW, we STRONGLY suggest (just to be doubly sure) that you encrypt at rest and/or offer the option to use Customer Supplied Keys. Encrypt all the things! https://cloud.google.com/security/encryption-at-rest/
I can't speak for Netflix's privacy policy, but, if your concern is with Google, we never ever ever ever mix/merge Google customer data with customer data. Like ever ever ever.
FWIW, we STRONGLY suggest (just to be doubly sure) that you encrypt at rest and/or offer the option to use Customer Supplied Keys. Encrypt all the things! https://cloud.google.com/security/encryption-at-rest/
To whom? One would imagine that the viewing data they collect would be one of their advantages over their competitors, making it detrimental to their business to sell it?
I'm guessing there's some project getting underway to move/duplicate some subset of resources hosted in AWS on GCP. It's likely a huge project that dwarfs most corporations entire cloud presence, but I'm guessing for Netflix it's a small piece of a giant puzzle, and that the vast majority of their operations are still centered at AWS and will be for years and years to come.