How much is Spotify paying Google Cloud?(medium.com)
medium.com
How much is Spotify paying Google Cloud?
https://medium.com/@davidmytton/how-much-is-spotify-paying-google-cloud-ebb3bf180f15
87 comments
Batching doesn't make any difference to the pricing:
> Batched calls are accounted for as multiple operations.
https://cloud.google.com/pubsub/pricing
> Batched calls are accounted for as multiple operations.
https://cloud.google.com/pubsub/pricing
Note by "queuing and batching" Spotify may be aggregating events client-side. Given that messages are priced in 64K increments, it would be most cost efficient to to get each message as close to 64K as possible.
i.e.
i.e.
# serialized: 1 message, 1 op
publish("['foo', 'bar']")
# instead of:
# a) 2 messages, 2 ops
publish("foo")
publish("bar")
# b) 2 messages, 1 call, 2 ops
publish("foo", "bar")Yes then it would be down to their consumers to split them out. However, the graphs Spotify show from Google's monitoring do not support this and they show the actual volume of messages as quoted in the article.
I agree that the current graphs from Spotify don't support this. They do seem to show a one-to-one mapping between events and messages emitted. Given the comparison in my GP, I think that client-side message collation/aggregation would be a natural next step.
I think what people are proposing is that instead of:
64 1k messages for Events, sent individually or in a batch
1 64k message for BatchedEventThat doesn't match the graphs Spotify post in their blog post. If the batch gets split out into individual operations then Google still prices for each message and each message gets delivered to the consumer. If it's just a single message that'd show up as a single message as far as Google is concerned, which is not what the Spotify test is showing.
Yes, but if I'm reading their blog post correctly, their individual messages are much smaller than 64K
"Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."
"Pub/Sub passed the test with flying colours. We published 2M messages without any service degradation and received almost no server errors from the Pub/Sub backend. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub."
Yes, but that has no impact on the cost. It's a flat rate in 64kb blocks. Batching and compression would only affect the networking fees from Google, not the per API call fees. If you batch up 64 1kb messages into a single batch then Google still bills you for 64 publish API calls.
They are wrapping small events into fewer larger ones. Hence the word batching.
I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!
I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!
"Batched calls are accounted for as multiple operations." - https://cloud.google.com/pubsub/pricing
Right.
If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit.
But rather than publishing
{ "messages" : [ { message 1 }, { message 2 } ] }
You publish
{ "messages" : [ { container message containing both, still less than 64k } ] }
Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is.
If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.
If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit.
But rather than publishing
{ "messages" : [ { message 1 }, { message 2 } ] }
You publish
{ "messages" : [ { container message containing both, still less than 64k } ] }
Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is.
If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.
He's talking about a single call containing multiple messages in their own serialization format.
[deleted]
If 2M requests result in 1 Gb network traffic, each message is 500 bits (~60 bytes). Theoretically, that's 1K messages per batch.
Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.
Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.
Batching could mean they're aggregating calls prior to handing then off to google as a single request. If you have 64k minimum block size (guessing) you can stuff in many small requests before sending them off.
Right but that only affects networking costs, because the batch is split out into individual publish API calls on Google's side, and that's what you're billed for.
[deleted]
$290k per month is $3.5M per year.
A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k.
This means that according to this estimation, Spotify pays for 20 Google software engineers.
[1] https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm [2] http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...
A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k.
This means that according to this estimation, Spotify pays for 20 Google software engineers.
[1] https://www.glassdoor.com/Salary/Google-Salaries-E9079.htm [2] http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...
> A software engineer at Google earns, on average, $127k per year [1]. We multiply that by 1.4 [2] to obtain a cost per employee of $177k.
That ignores RSUs, signing bonus, and other bonuses, which make up the majority of the compensation package for senior engineers.
Even at entry level, total comp (ignoring insurance, cost of office space, free food, and other things that aren't directly paid to the employee) is over $127k/yr. A baseline new grad offer last fall included $180k of stock, or $45k/yr, on top of a six figure salary plus a $50k signing bonus. This probably goes without saying, but new grad offers can go significantly higher if the new grad has strong competing offers, and senior offers start out even higher and have a much larger range.
That ignores RSUs, signing bonus, and other bonuses, which make up the majority of the compensation package for senior engineers.
Even at entry level, total comp (ignoring insurance, cost of office space, free food, and other things that aren't directly paid to the employee) is over $127k/yr. A baseline new grad offer last fall included $180k of stock, or $45k/yr, on top of a six figure salary plus a $50k signing bonus. This probably goes without saying, but new grad offers can go significantly higher if the new grad has strong competing offers, and senior offers start out even higher and have a much larger range.
So I don't get this. 1gbps and 2 million events/s could be comfortably handled on a single beefy 10gbps server. Add a second for failover. That's maximum $20K/year which is far cry from $3.5 million. How does this even begin to make economic sense? The economy of the cloud has never made sense to me, since it's cheaper to overbuy your capacity by 5x in dedicated machines than use AWS or GCE. After just one year you've paid for all your capital expenditures and then you're minting it. For a startup the appeal of the cloud is obvious, but for a big established company with known technical requirements I don't get it at all.
You think like an engineer :-)
I think cloud services are appealing for a couple of reasons from a top-brass perspective.
a) You free the capital, and the balance sheet looks nicer.
b) It's less complicated on a PowerPoint level. Just a box. (probably not the reason Spotify opted for google though.)
c) You don't think it's your core competence to keep servers safe and running, and rather would like to spend the time and effort on developing services while enjoying reason a) although you know it won't be cheaper.
d) Maybe it's hard to find the right people
I think cloud services are appealing for a couple of reasons from a top-brass perspective.
a) You free the capital, and the balance sheet looks nicer.
b) It's less complicated on a PowerPoint level. Just a box. (probably not the reason Spotify opted for google though.)
c) You don't think it's your core competence to keep servers safe and running, and rather would like to spend the time and effort on developing services while enjoying reason a) although you know it won't be cheaper.
d) Maybe it's hard to find the right people
I don't think he thinks like an engineer (not to insult him), but I think he's overlooking the challenges that exist to running large-scale systems.
> Add a second for failover.
That's not how it works. How does the second server pick up where the first server left off, when the requests are stateful? How does it know when to take over? (What detects that and how?) Or if it's a human, how does that human make the judgment call?
Cloud services are absolutely appealing from an engineering perspective. I am capable of building an application and running it on a fleet of machines such that the infrastructure takes care of itself completely and 100% by itself. Essentially no failure mode can involve me, unless either (A) my application has a defect in its logic, in which case it's unavoidable under any solution (B) probably a lot of customers are similarly affected, and someone is working on it. The data stores that I use are hosted and will not be impacted by any typical failure mode. The application that I've designed is deployed onto machines automatically, and if any of them encounter a failure that takes them offline, they'll be replaced automatically in a way that's a non-event. Furthermore, I have a blueprint for such systems such that building a new application in this way requires no more effort than filling out a single web form, after which an automation layer sets everything up, from the source control to the continuous build and deployment onwards.
Not all systems are simple enough to fit into this model, but if you account for it while designing new software, quite a lot of it can be. With the right building blocks you can operate very operations-free infrastructure at the application layer even at large scale. These problems might start to look different once you consider challenges that require tens, hundreds, thousands, or more individual machines to solve.
Another challenge is ensuring that new systems are built in a way from the beginning that can scale and that will be easy for other people to maintain. You don't want some random new-hire at your company setting up the "opaque box from hell" that has all the data on its disk (and, ahem, a "second for failover", that through a mysterious and unknown mechanism somehow takes over for the first) and that no one else will be able to maintain during failure. You also don't want to be surprised a year or two down the line when you're scaling up for the Super Bowl or Tax Day or Black Friday, or whatever, and your software falls over because "Bob the Intern, who thought every software system can be solved with a single box and 100 lines of assembly code" didn't anticipate that the business would be larger one day. If you standardize onto a certain set of building blocks, you can build software that, in addition to being operations-free, can scale to a large degree.
Note: I'm speaking hyperbolically. No software is truly operations-free. It's all a matter of degree. I have worked however with teams that are expected to hand-manage their own machines, and teams that use effective cloud automation, and the pattern I've seen is a huge difference in their operational load from routine engineering tasks. I have also never seen systems that completely 100% automatically scale themselves, but there is still a huge difference between systems where you simply say "I want 10 machines" / "I want 10 requests per second to my data store", and a system like, "I guess it's time to tell our DBA to figure out how to shard our MySQL server".
I also think that a large company could achieve a lot of the benefits provided by the cloud in-house, with its own data centers, if it had really strong internal automation and APIs, and strong teams who operate shared services for other teams (e.g., managed data layers). The really strong software I'm aware of for solving these problems has been kept as a competitive advantage, not released open source. So the practical reality is that few companies have through-and-through automation of the quality and degree that you can get from top cloud providers.
> a) You free the capital, and the balance sheet looks nicer.
Yes, I agree that this is a genuine reason for some companies to prefer the cloud, but these are largely not companies who have "top brass". Companies that are large enough to be "top brass" are typically not so capital-constrained as that this would majorly affect their purchasing decisions. The companies that care about this, rather, tend to be smaller and more capital-constrained.
I think the more mundane reason why companies prefer the cloud, which is frequently overlooked, is that individual business units can purchase cloud services with more autonomy than they could traditionally order IT services in-house. Cloud budgets often fit within what an executive can expense within their own authority on a month-to-month basis, where as the same capital purchase would require more review and approval. So some companies tend toward the cloud because it makes them more agile by providing decisions they can make that are smaller and more easily fit within the scope of less-senior people.
> Add a second for failover.
That's not how it works. How does the second server pick up where the first server left off, when the requests are stateful? How does it know when to take over? (What detects that and how?) Or if it's a human, how does that human make the judgment call?
Cloud services are absolutely appealing from an engineering perspective. I am capable of building an application and running it on a fleet of machines such that the infrastructure takes care of itself completely and 100% by itself. Essentially no failure mode can involve me, unless either (A) my application has a defect in its logic, in which case it's unavoidable under any solution (B) probably a lot of customers are similarly affected, and someone is working on it. The data stores that I use are hosted and will not be impacted by any typical failure mode. The application that I've designed is deployed onto machines automatically, and if any of them encounter a failure that takes them offline, they'll be replaced automatically in a way that's a non-event. Furthermore, I have a blueprint for such systems such that building a new application in this way requires no more effort than filling out a single web form, after which an automation layer sets everything up, from the source control to the continuous build and deployment onwards.
Not all systems are simple enough to fit into this model, but if you account for it while designing new software, quite a lot of it can be. With the right building blocks you can operate very operations-free infrastructure at the application layer even at large scale. These problems might start to look different once you consider challenges that require tens, hundreds, thousands, or more individual machines to solve.
Another challenge is ensuring that new systems are built in a way from the beginning that can scale and that will be easy for other people to maintain. You don't want some random new-hire at your company setting up the "opaque box from hell" that has all the data on its disk (and, ahem, a "second for failover", that through a mysterious and unknown mechanism somehow takes over for the first) and that no one else will be able to maintain during failure. You also don't want to be surprised a year or two down the line when you're scaling up for the Super Bowl or Tax Day or Black Friday, or whatever, and your software falls over because "Bob the Intern, who thought every software system can be solved with a single box and 100 lines of assembly code" didn't anticipate that the business would be larger one day. If you standardize onto a certain set of building blocks, you can build software that, in addition to being operations-free, can scale to a large degree.
Note: I'm speaking hyperbolically. No software is truly operations-free. It's all a matter of degree. I have worked however with teams that are expected to hand-manage their own machines, and teams that use effective cloud automation, and the pattern I've seen is a huge difference in their operational load from routine engineering tasks. I have also never seen systems that completely 100% automatically scale themselves, but there is still a huge difference between systems where you simply say "I want 10 machines" / "I want 10 requests per second to my data store", and a system like, "I guess it's time to tell our DBA to figure out how to shard our MySQL server".
I also think that a large company could achieve a lot of the benefits provided by the cloud in-house, with its own data centers, if it had really strong internal automation and APIs, and strong teams who operate shared services for other teams (e.g., managed data layers). The really strong software I'm aware of for solving these problems has been kept as a competitive advantage, not released open source. So the practical reality is that few companies have through-and-through automation of the quality and degree that you can get from top cloud providers.
> a) You free the capital, and the balance sheet looks nicer.
Yes, I agree that this is a genuine reason for some companies to prefer the cloud, but these are largely not companies who have "top brass". Companies that are large enough to be "top brass" are typically not so capital-constrained as that this would majorly affect their purchasing decisions. The companies that care about this, rather, tend to be smaller and more capital-constrained.
I think the more mundane reason why companies prefer the cloud, which is frequently overlooked, is that individual business units can purchase cloud services with more autonomy than they could traditionally order IT services in-house. Cloud budgets often fit within what an executive can expense within their own authority on a month-to-month basis, where as the same capital purchase would require more review and approval. So some companies tend toward the cloud because it makes them more agile by providing decisions they can make that are smaller and more easily fit within the scope of less-senior people.
>individual business units can purchase cloud services with more autonomy
Yes, this is also probably true, as hardware costs are typically very noticeable as they are large but few. The old trick of splitting the bill works fine :-)
Anyway, I'm a control freak. I would very much want my teams to keep control of everything that is important to our core product and services, including servers and infrastructure - and everything that have to be ordered from another party - internal or not, actually more so if it's inhouse, is a potential delay or outage that is not fixed as soon as it is possible.
But then again, everyone should know their limits... :-)
Yes, this is also probably true, as hardware costs are typically very noticeable as they are large but few. The old trick of splitting the bill works fine :-)
Anyway, I'm a control freak. I would very much want my teams to keep control of everything that is important to our core product and services, including servers and infrastructure - and everything that have to be ordered from another party - internal or not, actually more so if it's inhouse, is a potential delay or outage that is not fixed as soon as it is possible.
But then again, everyone should know their limits... :-)
ISP cost, backups, patching, hardware upgrades, co-location, DR, insurance, paying for physical and cyber-security, personnel etc.
outsourcing allot of those things saves money it's also a different expense of you buy infrastructure it has value which means depreciation overtime which affects how your accounting is done.
Cloud makes sense for allot of companies not just startups the biggest reason why companies with eatablished infrastructure don't migrate is the initial cost and regulation.
outsourcing allot of those things saves money it's also a different expense of you buy infrastructure it has value which means depreciation overtime which affects how your accounting is done.
Cloud makes sense for allot of companies not just startups the biggest reason why companies with eatablished infrastructure don't migrate is the initial cost and regulation.
Why can't you just rent the server from some company? Such a company is only responsible for making sure the hardware works (there are no power outages, the disks work, the network is up and reachable) and you have to make sure your application works and have some failover (because the hardware will eventually fail and they will only replace it, not make up for the loss).
You'd still save a lot of money compared to cloud servers, need little admin work (the same compared to compute instances – amazon's EC2 servers), and have a reliable server. Providing the servers doesn't take forever, either. Some bare metal servers are up and running within 120sec, though, you can't just shut them down and stop paying like cloud servers.
Companies that offer these services are OVH.com and softlayer, though softlayer always seemed to be a bit expensive. I'm sure you'll be able to find similar companies though.
I can see that this makes little sense to startups because the admin work is way more expensive but for bigger companies, this should not be the case (I don't consider Spotify and Netflix a startup anymore).
You'd still save a lot of money compared to cloud servers, need little admin work (the same compared to compute instances – amazon's EC2 servers), and have a reliable server. Providing the servers doesn't take forever, either. Some bare metal servers are up and running within 120sec, though, you can't just shut them down and stop paying like cloud servers.
Companies that offer these services are OVH.com and softlayer, though softlayer always seemed to be a bit expensive. I'm sure you'll be able to find similar companies though.
I can see that this makes little sense to startups because the admin work is way more expensive but for bigger companies, this should not be the case (I don't consider Spotify and Netflix a startup anymore).
Why can't you just rent the server from some company? Such a company is only responsible for making sure the hardware works (there are no power outages, the disks work, the network is up and reachable)
Isn't that what Amazon does? (even though they are virtual servers)?
You'd still save a lot of money compared to cloud servers,
Can you post some numbers? A 24 core 48GB Dell server with 3x120GB SSD's costs around $11K, or around $300/month for a 36 month lease.
An 8 core 15GB 160GB AWS c3.2xlarge costs around $200 each per month with a 3 year reserved instance. 3 of them will fit in one of the Dells, so it would cost around $600/month, or about $300 more than the raw purchase price for the hardware alone.
But also with that price you get network, power, cooling, rack space, maintenance, etc, so it's not clear that buying and running that server is significantly cheaper.
Some bare metal servers are up and running within 120sec
How long does it take if you want 500 of them? And 2 hours later you want to turn them off. Then 2 hours after that you want to bring them all back online again.
you can't just shut them down and stop paying like cloud servers.
But that's the big value-add in Cloud servers, it's capacity on demand so you only pay for what you us -- need to spin up 500 servers to run month-end processing? You only need to pay for them for the few hours or days a month when you need them.
Further, you can add multi-datacenter or multi-region redundancy at very little additional cost - you don't need to keep those redundant servers running when you're not using them.
Isn't that what Amazon does? (even though they are virtual servers)?
You'd still save a lot of money compared to cloud servers,
Can you post some numbers? A 24 core 48GB Dell server with 3x120GB SSD's costs around $11K, or around $300/month for a 36 month lease.
An 8 core 15GB 160GB AWS c3.2xlarge costs around $200 each per month with a 3 year reserved instance. 3 of them will fit in one of the Dells, so it would cost around $600/month, or about $300 more than the raw purchase price for the hardware alone.
But also with that price you get network, power, cooling, rack space, maintenance, etc, so it's not clear that buying and running that server is significantly cheaper.
Some bare metal servers are up and running within 120sec
How long does it take if you want 500 of them? And 2 hours later you want to turn them off. Then 2 hours after that you want to bring them all back online again.
you can't just shut them down and stop paying like cloud servers.
But that's the big value-add in Cloud servers, it's capacity on demand so you only pay for what you us -- need to spin up 500 servers to run month-end processing? You only need to pay for them for the few hours or days a month when you need them.
Further, you can add multi-datacenter or multi-region redundancy at very little additional cost - you don't need to keep those redundant servers running when you're not using them.
Just a side note. After some benchmarking, I recently discovered that c3.2xlarge does not actually have 8 cores, rather it had 4 real cores = 8 hyper threaded cores. I was very surprised by this and suspect most other AWS users would also be surprised. This is not to say that AWS is a bad deal, but I think Amazon could be a little more transparent about it.
See http://www.pythian.com/blog/virtual-cpus-with-amazon-web-ser... for more details.
See http://www.pythian.com/blog/virtual-cpus-with-amazon-web-ser... for more details.
a bigger issue than the cost, even the fully loaded cost is the capability you can buy.
I have a feeling that a lot of faults that in the google infrastructure result in 2 minutes down time, or even just a raised eyebrow in the ops centre would put most SME's (or event big corporates) on there arse for a week. Do you have a spare generator? If main power goes how long do the UPS keep you going for ? If you have a dc fire what happens... These are things that you can plan for but need real capability to deal with, capability that needs scale.
I have a feeling that a lot of faults that in the google infrastructure result in 2 minutes down time, or even just a raised eyebrow in the ops centre would put most SME's (or event big corporates) on there arse for a week. Do you have a spare generator? If main power goes how long do the UPS keep you going for ? If you have a dc fire what happens... These are things that you can plan for but need real capability to deal with, capability that needs scale.
Why should you? also you forget that not ever service scales the same way or has the same usage across the year.
Take for example something old school and established (ZzZzZ) - accounting.
Say you are an accounting firm with an established IT infrastructure and you now want to launch a new service that will allow new smaller clients to perform certain tasks even a single one (say fixed assets devaluation for deductibles). The core of the service is a messaging API as it's an accounting application it's usage wont be fixed and the product team estimates that on a normal week it will handle 20,000 requests, on a end or the quarter week it will handle 500,000 and at the end of the financial year it will handle 2,000,000 how do you scale this?
Say a single server will 20,000 request so you have here a 1 to 100 scaling between peak and normal usage which means that if you own your own infrastructure you are paying for 99 servers to idle during most of the time. Even if you can some how manage to do some inner-organizational thin provisioning you will still be wasting allot of hardware because as it turns out most of your other internal services also follow a similar usage trend so that might not be an option to begin with.
Now you can rent VPS/IAAS from say Amazon or Rackspace but then you still have a huge overhead sure you'll pay less for usage because those 99 instances will be down but you still need to pay for other resources e.g. reserved IP addresses and storage, you need to practice and test the machines every couple of weeks/months and you need an live-ops/sys-admins with balls of steel to be confident enough to agree to spin up 99 instances at the very last moment and hope that everything will work out smoothly.
But even that doesn't really solve it because it doesn't allow you to be flexible with onboarding new costumers, it can't predict various trends like a change in legislation or a holiday weekend prompting people to use your service earlier / later than expected.
But if you say just use a PAAS/SAAS messaging service all of this goes away, you don't worry about scaling as long as the service can handle your peak traffic, you don't worry about waste because you pay what you use so if you've only sent 18796 messages in week 37 you pay for that exact amount, you don't need to worry about some promotion that the marketing and product dev team was running that caused an influx of new customers without telling you, and you don't have to really worry about down time, backups, and even security (as long as you get sufficient assurance from your service provider) because as long as their service is up you are up and running.
Take for example something old school and established (ZzZzZ) - accounting.
Say you are an accounting firm with an established IT infrastructure and you now want to launch a new service that will allow new smaller clients to perform certain tasks even a single one (say fixed assets devaluation for deductibles). The core of the service is a messaging API as it's an accounting application it's usage wont be fixed and the product team estimates that on a normal week it will handle 20,000 requests, on a end or the quarter week it will handle 500,000 and at the end of the financial year it will handle 2,000,000 how do you scale this?
Say a single server will 20,000 request so you have here a 1 to 100 scaling between peak and normal usage which means that if you own your own infrastructure you are paying for 99 servers to idle during most of the time. Even if you can some how manage to do some inner-organizational thin provisioning you will still be wasting allot of hardware because as it turns out most of your other internal services also follow a similar usage trend so that might not be an option to begin with.
Now you can rent VPS/IAAS from say Amazon or Rackspace but then you still have a huge overhead sure you'll pay less for usage because those 99 instances will be down but you still need to pay for other resources e.g. reserved IP addresses and storage, you need to practice and test the machines every couple of weeks/months and you need an live-ops/sys-admins with balls of steel to be confident enough to agree to spin up 99 instances at the very last moment and hope that everything will work out smoothly.
But even that doesn't really solve it because it doesn't allow you to be flexible with onboarding new costumers, it can't predict various trends like a change in legislation or a holiday weekend prompting people to use your service earlier / later than expected.
But if you say just use a PAAS/SAAS messaging service all of this goes away, you don't worry about scaling as long as the service can handle your peak traffic, you don't worry about waste because you pay what you use so if you've only sent 18796 messages in week 37 you pay for that exact amount, you don't need to worry about some promotion that the marketing and product dev team was running that caused an influx of new customers without telling you, and you don't have to really worry about down time, backups, and even security (as long as you get sufficient assurance from your service provider) because as long as their service is up you are up and running.
+1
Not to mention that often it can takes weeks for new hardware order to be approved / delivered / provisioned.
Cloud? Just dial it up... Well, if you are at Netflix scale, you probably couldn't just dial it up, and would have to work with the vendor to ensure adequate capacity was available for your use.
Not to mention that often it can takes weeks for new hardware order to be approved / delivered / provisioned.
Cloud? Just dial it up... Well, if you are at Netflix scale, you probably couldn't just dial it up, and would have to work with the vendor to ensure adequate capacity was available for your use.
And keep in mind that adding those new servers even in managed eviromnents can fail and fail in unexpected ways like taking out your whole network. We had this happen with rackspace - hence we moved to Amazon
If a single point of failure took down your entire network, then your solution wasn't engineered properly. Just because on provider messed up big time doesn't mean every other provider would. The hardwdare design/architecture can also be provided to you for free as part of the sales process, no extra service required.
As for hardware failures in general, you can afford to have some complete extra servers on hand doing nothing most of the time with the amount of money you'd save on renting bare metal instead of paying for cloud. Or if engineered properly, you'd be using all the machines at once but be able to afford to lose half of them, and failover automatically.
Yes, a cloud (bunch of VMs on a cluster) is going to be more reliable than a single machine, as you get hardware redundancy. But a cluster of machines with hardware redundancy is going to be much more reliable than a bunch of VMs on a cluster, as there's less complexity involved without the cloud stack, and thus fewer points of failure.
As for hardware failures in general, you can afford to have some complete extra servers on hand doing nothing most of the time with the amount of money you'd save on renting bare metal instead of paying for cloud. Or if engineered properly, you'd be using all the machines at once but be able to afford to lose half of them, and failover automatically.
Yes, a cloud (bunch of VMs on a cluster) is going to be more reliable than a single machine, as you get hardware redundancy. But a cluster of machines with hardware redundancy is going to be much more reliable than a bunch of VMs on a cluster, as there's less complexity involved without the cloud stack, and thus fewer points of failure.
In our case, I mostly agree - we however were led to believe we had network redundancy and when they added new servers and it effectively blacked out access to all servers we realized this was not the case. My realization at the time was simple. It's better to be on a ship with a lot of engineers dedicating their time to a bigger system then hoping an engineer finds the time to check on your one off raft managed solution.
Public clouds are an efficiency gain for both the datacenter and the end user. The bottom line to me is more engineers are focused on operating and improving the public cloud than the state of the art managed dedicated racks from 2014.
Public clouds are an efficiency gain for both the datacenter and the end user. The bottom line to me is more engineers are focused on operating and improving the public cloud than the state of the art managed dedicated racks from 2014.
On a balance sheet, you are trading CapEx for OpEx. That's it. For certain organizations with specific types of computing loads, that may make the most sense.
(btw -- I disagree with your math there on the actual cost -- I agree it could ultimately be cheaper to run things in house, there's no way it's going to go from $3.5 million to $20k annually. It'd be more like $3.5 million to $2 million annually. At that point, it's a question of whether or not you want the opportunity cost of having your tech staff engaged in deploying/operating/maintaining server infrastructure as opposed to building new features/etc)
(btw -- I disagree with your math there on the actual cost -- I agree it could ultimately be cheaper to run things in house, there's no way it's going to go from $3.5 million to $20k annually. It'd be more like $3.5 million to $2 million annually. At that point, it's a question of whether or not you want the opportunity cost of having your tech staff engaged in deploying/operating/maintaining server infrastructure as opposed to building new features/etc)
You might be underestimating just how valuable it is to make things somebody else's problem. More employees means more real estate, regulations, taxes, recruiters, HR drones, lawyers, personnel issues, bureaucratic decay, and all of the other challenges of running a company.
And if Spotify wants to change strategy tomorrow, they can just end their contract with Google. Winding down internal operations is way more complicated.
And if Spotify wants to change strategy tomorrow, they can just end their contract with Google. Winding down internal operations is way more complicated.
Virtualizing your server resources is essentially outsourcing a whole lot of sysadmin work to a vendor.
Dedicated servers create a different kind of lock-in: Dynamically scaling up dedicated servers is like laying down the tracks while the train is moving. Acceptable to some, but not to a company like Spotify.
Dedicated servers create a different kind of lock-in: Dynamically scaling up dedicated servers is like laying down the tracks while the train is moving. Acceptable to some, but not to a company like Spotify.
You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique. Physical hardware is actually quite reliable and if you follow the same practices you should in the cloud - namely automatic failover, eliminate single points of failure, etc then the differences decrease.
Dynamically scaling load is only really interesting if the load is extremely variable. You could just buy 5x the dedicated hardware, which will likely be able to serve 10x the traffic and may still end up paying less. Who cares if it sits idle most of the time.
Now Spotify, like Netflix probably has extreme variance in load, much more so than a normal company. So maybe the cloud works for them. But that's not a common situation at all.
Dynamically scaling load is only really interesting if the load is extremely variable. You could just buy 5x the dedicated hardware, which will likely be able to serve 10x the traffic and may still end up paying less. Who cares if it sits idle most of the time.
Now Spotify, like Netflix probably has extreme variance in load, much more so than a normal company. So maybe the cloud works for them. But that's not a common situation at all.
It's definitely common for IT staffing costs to increase rather than decrease with a cloud transition, at least so far. Part of the problem is that you often are trading off sysadmins (relatively cheap) for engineers (relatively expensive), as you reengineer your system to run on top of ephemeral, distributed cloud servers, with new failure cases and new technology stacks. This is often justified on the basis that a large chunk is one-time engineering cost that will pay for itself in the long run. But that's a speculative bet, especially considering how much technology churn is happening so far (e.g. a bunch of companies that invested heavily in early 2010s cloud stacks are now re-investing in Docker). It's possible things will stabilize enough for such companies to eventually lay off many of their distributed-systems engineers and devops staff, and go more into cloud-autopilot mode with low IT headcount and lower staffing cost than pre-cloud-transition. But I don't think many companies are there yet, and I wouldn't bet a lot of money on it myself.
I work for a company (Pivotal) where a major revenue source is selling a packaged version of Cloud Foundry.
Our customers like it because their existing ops staff can greatly reduce their workload and allow development staff to move much faster.
Our customers like it because their existing ops staff can greatly reduce their workload and allow development staff to move much faster.
>You outsource some of the sysadmin work, but not all of it. Cloud servers have plenty of their own failure cases, some of them unique.
The thing with them is if they go down you have Google to worry just as much as you (if not more) for bringing them back up.
You don't have that with your own servers.
The thing with them is if they go down you have Google to worry just as much as you (if not more) for bringing them back up.
You don't have that with your own servers.
Google and their famous customer support, yes. Worried they might be, but more than you? For them, the cloud is one of many revenue streams, and nowhere near their most profitable. For you-cloud-customer, a website outage might mean you don't have a revenue stream anymore. It could make the difference between making payroll and shutting down next week. I bet you'll be much more worried than Google will ever be.
Googlers might be more skilled than your ops team, yeah, but I would never bet on huge companies to worry about the fate of something that is not their main revenue generator.
Googlers might be more skilled than your ops team, yeah, but I would never bet on huge companies to worry about the fate of something that is not their main revenue generator.
It's for the same reason MSPs were a thing in the 90s. People simply don't want to screw with machines or the people who run them. Given Spotify's use case isn't that security sensitive, it makes perfect sense to let someone else worry about keeping your machines up.
In short, it's insurance.
In short, it's insurance.
Redundancy and spare capacity everywhere in production, staging environment, QA environment, development environments ...
Remember that at Google, like in most other big companies in the Valley, baseline salary is only one part of total compensation: stock grants and bonuses are also quite significant.
On the other hand, this blog only estimates the cost for Pub/Sub, when Spotify is using a whole slew of other Google Cloud services as well.
On the other hand, this blog only estimates the cost for Pub/Sub, when Spotify is using a whole slew of other Google Cloud services as well.
It'll be interesting to see what other numbers Spotify release in subsequent blog posts for the other services they're using. Hopefully some cost calculations can be run for those, too.
Note that this is an estimate for just one feature of Spotify, not of their total hosting bill.
[deleted]
> This means that according to this estimation, Spotify pays for 20 Google software engineers.
If hardware were free, sure.
If hardware were free, sure.
Note, google cloud requires hardware.
As far as I've heard from Netflix's end the bottom line cost was not as big a factor in their infrastructure decisions as their content contracts were a much larger % of their budget. I suspect Google wants to highlight more the fact that it works and works well rather than highlight cost competitiveness at the moment.
thanks for the post @dmytton.
not sure if it's accurate It'd be lovely for Spotify or Google to chime in.
thanks for the post @dmytton.
not sure if it's accurate It'd be lovely for Spotify or Google to chime in.
My guess is they won't. I suspect that Google heavily subsidized the deal. One way to get market traction is to get market awareness, and Spotify on GCP achieves that. Spotify would have been dumb not to get a huge discount in exchange for its logo.
However, Spotify still stores its music files on AWS and uses CloudFront for content distribution. They're only gonna use Google Cloud for analyzing customer data. They won't ditch AWS...
http://arstechnica.co.uk/information-technology/2016/02/spot...
http://arstechnica.co.uk/information-technology/2016/02/spot...
This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was always full. They now have to pay for the storage and processing costs for that. How much will they pay? I don't know, but I know from their history they are very good at getting discounts on infrastructural software (e.g., their Hadoop software was licensed from Hortonworks and they got an incredible price, because Hortonworks needed a reference customer - sound familiar?)
I could only write about pub/sub in this article because that's the only service we have a full enough set of numbers for. I hope Spotify will be releasing more stats in their future blog posts about the other GCP services they'll be using.
I have the stats from late January (at Stockholm Hadoop User Group). It's 90PB of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000 CPUs with probably 60-90% utilization).
I should add they also have ~100 servers streaming the music to clients for songs that aren't handled by the CDN.
I am wondering, how does one migrate huge amounts of data like this?
Sometimes a truck full of hard drives is the fastest/cheapest way. Not saying that's what they did, but it has been done before.
Yes, that's how the England 100,000 genomes project is transferring 10s of petabytes from San Diego to England. I assume it was the same here.
If any of the data to be migrated is music then much of that is probably somewhere inside google's datacenters already..
The author also failed to notice "peaks" from the original spotify blog post about events per second. So 700k requests is not average, it's peak. They don't have that every second.
This is a good point, but it could be somewhat mitigated by their expected growth of up to 2M per second. To quote the full line from their blog post:
> Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.
In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
> Currently our production load peaks at around 700K events per second. To account for the future growth and possible disaster recovery scenarios, we settled on a test load of 2M events per second.
In the absence of any other numbers, using the 700k number seems reasonable otherwise you would start to get into modelling without any real data to back up your assumptions. This at least gives you a maximum they would be paying under current production conditions.
That's actually a reasonable response. From my first read the numbers seemed a bit inflated, so I dug into their blog post out of curiosity (I develop on GAE), so the comment was not ill intended. Now that you have pointed out they do seem a good estimate!
Thanks for the great articles David, keep them coming! :)
Thanks for the great articles David, keep them coming! :)
If this is the logic, then a better title would have been "How much will Spotify be paying Google Cloud in a few years, assuming 3x growth from today's numbers?"
It's really not fair to tell the world that their costs are based on average of 700k rps, when that is their peak traffic through the system.
It's really not fair to tell the world that their costs are based on average of 700k rps, when that is their peak traffic through the system.
How about a gaussian distribution which peaks at 700k?
Assuming max as the norm is not too right. (just to make the number look high)
Assuming max as the norm is not too right. (just to make the number look high)
You could guess that, but with no real expectation that it's right either. E.g. for a global service I'd guess it's better viewed as a sum of three offset distributions, one per continent, but even then the individual distributions are unlikely to be gaussian.
Having run global services myself I can tell you that the shape is usually no standard distribution. Peak/trough depends on the exact usage demographics, but the one I have on hand has a trough of ~1/2 peak, and an average of ~3/4.
Having run global services myself I can tell you that the shape is usually no standard distribution. Peak/trough depends on the exact usage demographics, but the one I have on hand has a trough of ~1/2 peak, and an average of ~3/4.
Just curious: how does the real distribution look like, from your experience? Do you have any graphs by any chance?
nice point.
If you want to further refine you could probably estimate a large chunk of early morning US/EU time where traffic will be very light. maybe reduce the estimate by 1/3 or 1/4th to account for this.
If you want to further refine you could probably estimate a large chunk of early morning US/EU time where traffic will be very light. maybe reduce the estimate by 1/3 or 1/4th to account for this.
As a side note, $300k/mo is a pretty small amount of money, compared to the +$100MM/mo revenue Spotify brings in.
> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.
I am pretty sure it is also not unusual for average customers on AWS to get some discount, with some reserved instance committment). One such example is probably Redshift which is extremely expensive.
Would be nice to get an estimate quote for AWS but just for SQS I don't expect to be too expensive.
I am pretty sure it is also not unusual for average customers on AWS to get some discount, with some reserved instance committment). One such example is probably Redshift which is extremely expensive.
Would be nice to get an estimate quote for AWS but just for SQS I don't expect to be too expensive.
I'd be willing to bet they're paying almost $0 in the first year. Google REALLY wants a big name customer, so you have to think they gave this deal away for negative margin.
> Thankfully, Google’s pricing is very transparent and simple to work with, so we can use Spotify’s blog posts to calculate some list pricing for what Spotify might be paying Google.
The biggest way in which all such assumptions are likely to be wrong is that they overlook the likelihood that Spotify has been offered a potentially substantial discount in exchange for attempting to win their business, and to be an early adopter. Not that there's anything wrong with that, but it's something to take into account while evaluating what you think Spotify might be paying.
Many cloud service providers offer a discount to, for example, qualified startups participating in certain accelerators. In the same fashion, it would not be surprising for a provider to entice a major potential user and hopeful net promoter with a significant price reduction as well as other perks, such as more access to senior people for operations support, as well as influence on the roadmap.
So while I think this blog post could plausibly estimate what a typical customer with Spotify's usage pattern might pay, assuming the estimates are correct (I haven't checked the details), one cannot draw conclusions about what they're likely to actually be paying.
[Edit: I see now that the article recognizes this at the bottom. At the time I made my comment, it seemed to have been overlooked in the discussion here, where only one comment acknowledged it and the rest seemed to discuss the numbers as factual and face-value: https://news.ycombinator.com/item?id=11242166 ]
The biggest way in which all such assumptions are likely to be wrong is that they overlook the likelihood that Spotify has been offered a potentially substantial discount in exchange for attempting to win their business, and to be an early adopter. Not that there's anything wrong with that, but it's something to take into account while evaluating what you think Spotify might be paying.
Many cloud service providers offer a discount to, for example, qualified startups participating in certain accelerators. In the same fashion, it would not be surprising for a provider to entice a major potential user and hopeful net promoter with a significant price reduction as well as other perks, such as more access to senior people for operations support, as well as influence on the roadmap.
So while I think this blog post could plausibly estimate what a typical customer with Spotify's usage pattern might pay, assuming the estimates are correct (I haven't checked the details), one cannot draw conclusions about what they're likely to actually be paying.
[Edit: I see now that the article recognizes this at the bottom. At the time I made my comment, it seemed to have been overlooked in the discussion here, where only one comment acknowledged it and the rest seemed to discuss the numbers as factual and face-value: https://news.ycombinator.com/item?id=11242166 ]
The author covered this:
>>> Is Spotify really paying $290,518/month?
>>> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. Cost is important but using public cloud and especially managed services like pub/sub, the biggest benefit is in ease of deployment and not having to manage/scale everything.
>>> Is Spotify really paying $290,518/month?
>>> Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts. Cost is important but using public cloud and especially managed services like pub/sub, the biggest benefit is in ease of deployment and not having to manage/scale everything.
[deleted]
This was actually addressed in the article towards the end, and the part you quoted does say list price.
> Is Spotify really paying $290,518/month? Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.
> Is Spotify really paying $290,518/month? Of course, for such a high profile customer and for the priviledge of a case study, we can assume Spotify has some heavy discounts.
I recall David Byttow mentioning that secret cost about this much to run on GCP so I'm not surprised at all.
Yeah, Secret was spending a ton of money on GAE.
> For example, in one month, our server costs were well over $4M on Google App Engine after spiking 5M users in the course of two weeks. We scrambled and optimized, but the cost was real[1].
[1] - https://www.producthunt.com/live/david-byttow
> For example, in one month, our server costs were well over $4M on Google App Engine after spiking 5M users in the course of two weeks. We scrambled and optimized, but the cost was real[1].
[1] - https://www.producthunt.com/live/david-byttow
Is there a post talking about how much Netflix pays Amazon?
The post is a cool example but I'd be shocked if Spotify doesn't have a contract for up to X events/mo and overages at some reduced rate. No way they're paying per request. The only place I've never seen negotiate rates is Bloomberg.
The bigger question is if Google Cloud Platform is cheaper than AWS?
There must have been a steep discount given that Spotify has been an enormous reference customer and blogging about how much they love it.
A good guess using non-technical means would be $15-18M a year. I based this on the assumption that a customer doing less than $1M a month really isn't worth mentioning in the news or noteworthy. Also, Spotify did $1.2B last year.
P.S. Howdy David!
P.S. Howdy David!
Does Google cloud work in china?
The blog post says "We also added queuing and batching in the client. (This wasn’t first time that we needed to roll our sleeves up and reimplement a Google Cloud API client: in another project we implemented a high performance client for the Datastore API.)"