AWS kills Snowmobile data transfer truck(cnbc.com)
cnbc.com
AWS kills Snowmobile data transfer truck
https://www.cnbc.com/2024/04/17/aws-stops-selling-snowmobile-truck-for-cloud-migrations.html
81 comments
I'm not sure how much I can safely say, given I was employed by AWS, and provided some element of support to the snowball team from time to time, but Snowmobile was originally targeted towards one or two particular customers that had particular needs. Getting their data shifted was more than going to turn a profit on the whole exercise. Any other customers using the service was a bonus.
The amount of the data being shifted was one of the particular needs, but it was more the _other_ aspects of it being a huge truck that entered in to the equation.
The amount of the data being shifted was one of the particular needs, but it was more the _other_ aspects of it being a huge truck that entered in to the equation.
I wonder how much they pay for 100PB in AWS, depending on the storage service/tier. Also it will be quite expensive if they ever want to move out of AWS. Or is a project like this considered to be "eternal"?
Using the calculator (https://calculator.aws/#/createCalculator/S3), AWS S3 "infrequent access" is $1.3M per month for 100PB. Not sure if that's single zone.
If you put it in Glacier Deep Archive it’s closer to $1.3M per year, just to set a lower bound.
Deep Archive has retrieval times up to 12 hours, so for on-prem/colo that is then competing with buying a few tape robots (you can get a single tape archive that handles >1 exabyte, but you'll of course want redundancy without having to have staff shuffling tape in an archive that will run into the thousands). You can buy management software that tries to replicate a glacier-like experience for things like SpectraLogic tape libraries but they may not be the cheapest option.
It ends up becoming a very different beast than the storage classes w/fast enough retrieval guarantees to need disk, in any case.
It ends up becoming a very different beast than the storage classes w/fast enough retrieval guarantees to need disk, in any case.
Unlikely that they want it hot or even warm. That data almost definitely went to Gkacier while they worked out what to do with it.
You overestimate the enterprise.
It’s actually hard to say. The prices these companies pay can be very different than list prices for us mortals. It’s one of my biggest complaints about AWS; the prices seem high, but in reality they are very opaque.
This really needs to be made illegal already. All prices should be public and you should not be allowed to charge different customers different prices for the same automated service.
I should be able to store my data at the exact same $/TB price as Epic Games, for example. They once mentioned it costs them less than a penny to serve several GB from AWS - why the fuck is it more expensive for me?
I should be able to store my data at the exact same $/TB price as Epic Games, for example. They once mentioned it costs them less than a penny to serve several GB from AWS - why the fuck is it more expensive for me?
If you are willing to sign a contract and commit to spend $ millions per year on AWS services for 5+ years like Epic Games did, then sure, you can get the same $/TB price they got.
At some point of very large usage, you also have to work with AWS teams to give input on your demand forecast.
Unless you have such large usage you get list price, that also gives you the illusion of infinite elasticity.
This is not exclusive to AWS though, all hyperscalers have this policy.
At some point of very large usage, you also have to work with AWS teams to give input on your demand forecast.
Unless you have such large usage you get list price, that also gives you the illusion of infinite elasticity.
This is not exclusive to AWS though, all hyperscalers have this policy.
Every business negotiates volume discounts. Do you think consumer goods companies charge Walmart for products at the same rate that they charge a mom and pop shop?
I’m sure if you had the same volume as Netflix you would pay less than the retail price to.
If you ever flew on a plane, I can guarantee you that there are probably 20+ different prices being charged for the same seat.
I’m sure if you had the same volume as Netflix you would pay less than the retail price to.
If you ever flew on a plane, I can guarantee you that there are probably 20+ different prices being charged for the same seat.
Because you’re a more expensive customer per dollar of revenue, statistically - there are constant costs per customer for support, billing, etc.
Billing is automated. What support? S3 doesn't care whether it serves a PB of Epic traffic or from thousands of small customers, it's all automated. There is no actually reasonable explanation for the service being 10-100 times more expensive than greed.
Cards get declined. Support tickets get created. Customers die and you have to hold all their stuff open for 90 days while trying to reach them, without a chance of getting paid.
You might not be stupid, but there are people who look _just like you_ who will email support over a six cent billing issue which turns out to be correct anyway - but just burned a couple hours of support people’s time.
Epic probably doesn’t do stuff like that.
You might not be stupid, but there are people who look _just like you_ who will email support over a six cent billing issue which turns out to be correct anyway - but just burned a couple hours of support people’s time.
Epic probably doesn’t do stuff like that.
Many jurisdictions do require prices to be clearly displayed for goods, but services are a different beast. You have limited resources to deliver these services, customer needs can be complex, and differ across regions.
> This really needs to be made illegal already. All prices should be public and you should not be allowed to charge different customers different prices for the same automated service.
Why? Different people get different prices for products all the time. It's fundamental to a functioning free market.
Why? Different people get different prices for products all the time. It's fundamental to a functioning free market.
> They once mentioned it costs them less than a penny to serve several GB from AWS - why the fuck is it more expensive for me?
You spend far less money than a corporation, and there are customer costs beyond the actual bandwidth/compute charges.
You spend far less money than a corporation, and there are customer costs beyond the actual bandwidth/compute charges.
So what you are saying is that if I buy one Tesla, and you buy one hundred Tesla cars, they are not authorized to give you a discount?
Would it also be illegal to have 2 very similar services, with no small tier on one of them?
Why should it be illegal for service providers to negotiate prices?
they probably pay around 40% of list, depending on how much their ARR is.
The EU has forced the providers to allow a “final egress on account closing” at no cost (or i should say, they are offering it at no cost). GCP was first in January [0], the others have since matched.
0: https://www.theregister.com/AMP/2024/01/11/google_cloud_egre...
0: https://www.theregister.com/AMP/2024/01/11/google_cloud_egre...
Presumably doesn't apply to the cold archive tiers where a great deal of the cost is retrieval/egress?
the article lists the following for gcp: BigQuery, Cloud Bigtable, Cloud SQL, Cloud Storage, Datastore, Filestore, Spanner, and Persistent Disk.
It's also worth noting that the migration service only applies to data stored in one of the following services: BigQuery, Cloud Bigtable, Cloud SQL, Cloud Storage, Datastore, Filestore, Spanner, and Persistent Disk.
Still seems you would save a bundle if you had to pay to promote the data from archival to hot and then do the free exodus.
I would have expected we are close to steady state. The number of companies cyclically migrating too/from the cloud seems unlikely to decrease.
Sneakernets! https://en.m.wikipedia.org/wiki/Sneakernet
But I also never fully understood why an enterprise would pay so much money to give their data of this size to AWS to manage. It’s so much more expensive. If you’re at the capacity of needing a semi sneakernet, it would be far more cost effective and competitively advantageous to manage it yourself.
But I also never fully understood why an enterprise would pay so much money to give their data of this size to AWS to manage. It’s so much more expensive. If you’re at the capacity of needing a semi sneakernet, it would be far more cost effective and competitively advantageous to manage it yourself.
Maxar is a case that maybe makes sense. They have boatloads of satellite imagery that needs to be delivered to customers on demand.
Good luck achieving the realibility and security of AWS by managing it yourself at anything "competitive".
For some companies this would mean creating a whole department to manage that.
For some companies this would mean creating a whole department to manage that.
> For some companies this would mean creating a whole department to manage that.
If you are already managing that much data, you already have a whole department to manage that.
And in the end I haven't seen a significant decrease of people to manage stuff in the cloud. Sure you don't have to manage the hardware infra but you still manage tons of smaller virtual infrastructures on a myriad of AWS accounts. Sysadmins just get relabeled cloudops, devops and/or devsecops , end up being paid more, and thus more expensive.
That is at least my observation. I get paid more than I used to but the management hasn't been simplified. The only thing that has changed dramatically is you aren't dependent anymore on delivery time for storage, servers and network equipment.
If you are already managing that much data, you already have a whole department to manage that.
And in the end I haven't seen a significant decrease of people to manage stuff in the cloud. Sure you don't have to manage the hardware infra but you still manage tons of smaller virtual infrastructures on a myriad of AWS accounts. Sysadmins just get relabeled cloudops, devops and/or devsecops , end up being paid more, and thus more expensive.
That is at least my observation. I get paid more than I used to but the management hasn't been simplified. The only thing that has changed dramatically is you aren't dependent anymore on delivery time for storage, servers and network equipment.
> And in the end I haven't seen a significant decrease of people to manage stuff in the cloud.
This is a really important point. Virtually every customer that I work with who runs in the cloud has a platform group focused on cloud operation. They don't rack gear any more. Instead they do things like setting up IAM consistently that are beyond mortal ken. (Well mine, anyway.)
This is a really important point. Virtually every customer that I work with who runs in the cloud has a platform group focused on cloud operation. They don't rack gear any more. Instead they do things like setting up IAM consistently that are beyond mortal ken. (Well mine, anyway.)
Managing the data and managing the infrastructure to host that data are very different problems. As discussed elsewhere in this this thread you can store 100 petabytes of data in glacier for $1.2M/yr. That's literally the cost of 1-2 experienced engineers.
1. chance is that such amount of data is not sitting alone. It is online to be accessed. So you can't just factor glacier as a single cost.
2. a datacenter doesn't require many employees, most of them not being engineers themselves. To get back to my remark on point 1, while managing your own datacenters just to host storage would be expensive, it is usually mutualized with the rest of the infrastructure you need to access and use the data.
3. colo datacenters
2. a datacenter doesn't require many employees, most of them not being engineers themselves. To get back to my remark on point 1, while managing your own datacenters just to host storage would be expensive, it is usually mutualized with the rest of the infrastructure you need to access and use the data.
3. colo datacenters
That might be the cost of 1-2 experienced engineers in Silicon Valley. So don't host there. $1.2m is also for the Deep Archival storage class which has 12 hour retrieval times, and so competes with a handful of tape robots w/caching. It's not something you would have 1-2 experienced engineers spend full time managing - it's something you'd have remote hands and a support contract in place for, and your engineers spending a small fraction of their time managing.
With that much data you're paying a fortune for AWS Operations people either way. Every company I've seen that tries to large-scale AWS thing lean ends up driving themselves into a ditch where they spend millions on contractors, consultants and 3rd party vendors to dig themselves out of. And when they're in that ditch they are neither secure nor reliable.
Not if they use for long term storage or datasets, records, files and such.
That's if they use for infrastructure e.g. for some web service etc. And even then, it takes much less to use it, than to even begin designing such a solution.
That's if they use for infrastructure e.g. for some web service etc. And even then, it takes much less to use it, than to even begin designing such a solution.
I can set up an S3 bucket, efs, access, and backups, all reliable and geographically distributed in a single working day.
Yes, indeed. This is exactly the sneaky thing about the cloud. It seems so "simple".
Now try doing that for multiple multi-thousand person orgs with competing processes, architectures, product cycles, etc.
You'll find very quickly that costs spiral out of control in a very unpredictable and irreversible way.
Self-hosting is something a company has much tighter control of the costs for. Most of those costs are known up front as capex purchases and predictable opex expenditures. They tend to seem expensive compared to the PoC S3 demo you just did, but often aren't compared to the actual realized expenses after migrating.
Now try doing that for multiple multi-thousand person orgs with competing processes, architectures, product cycles, etc.
You'll find very quickly that costs spiral out of control in a very unpredictable and irreversible way.
Self-hosting is something a company has much tighter control of the costs for. Most of those costs are known up front as capex purchases and predictable opex expenditures. They tend to seem expensive compared to the PoC S3 demo you just did, but often aren't compared to the actual realized expenses after migrating.
… I have owned core product teams in multi thousand person companies and can confidently say that your spiraling costs sound like a lack of planning and product ownership. Platform teams are your friend.
Costs being well known for self hosting is just wrong. It completely ignores cost of ownership and tail risk for your infra
Costs being well known for self hosting is just wrong. It completely ignores cost of ownership and tail risk for your infra
You’re right. Because every company you have seen with a large scale implementation on AWS must mean that all large companies - including Netflix - are dumb.
I always find it funny that Netflix is considered a poster child of "all cloud all the time." If you look at Netflix's architecture, it actually uses the cloud in a very cost-effective way. They mostly deploy containers rather than using "cloudy" things like serverless, and Netflix's data plane is actually mostly their own CDN (Open Connect CDN), deployed in colo facilities around the world.
By data volume, I would guess that only 1% of Netflix's traffic (measured in gigabits per second) actually hits their cloud, if even that.
By data volume, I would guess that only 1% of Netflix's traffic (measured in gigabits per second) actually hits their cloud, if even that.
Just for reference: you are replying to someone who spent 3 years at AWS (Professional Services)….
I can tell you just from digging into the publicly available videos let alone some of the not publicly available information that you are greatly underestimating the complexity and costs of Netflix’s architecture.
You can even spend some time watching some of the interviews about Netflix’s transition from their own data centers to AWS and their reasons.
And before you accuse me of being a fanboy or shill. I got Amazoned last year. I have no love loss for Amazon as a former employee
I can tell you just from digging into the publicly available videos let alone some of the not publicly available information that you are greatly underestimating the complexity and costs of Netflix’s architecture.
You can even spend some time watching some of the interviews about Netflix’s transition from their own data centers to AWS and their reasons.
And before you accuse me of being a fanboy or shill. I got Amazoned last year. I have no love loss for Amazon as a former employee
I agree with you that Netflix's architecture in the cloud is incredibly complex. I know they do a lot of cool things with cloud storage, for example, but also scale breeds complexity. However, complexity is not scale.
Netflix users reportedly stream about 200 million hours per day, and at 1 GB/hour, that's an egress of 200 Petabytes/hour, or about 5 exabytes per day. There is no reason for Netflix to pay AWS to handle that, and they don't. That would be $100 million per day from CloudFront. You can read about their data plane online:
https://openconnect.netflix.com/en/
If you count how much data goes to their cloud vs their CDN appliances, I would be surprised if the cloud gets very much of their actual scale at all.
Netflix users reportedly stream about 200 million hours per day, and at 1 GB/hour, that's an egress of 200 Petabytes/hour, or about 5 exabytes per day. There is no reason for Netflix to pay AWS to handle that, and they don't. That would be $100 million per day from CloudFront. You can read about their data plane online:
https://openconnect.netflix.com/en/
If you count how much data goes to their cloud vs their CDN appliances, I would be surprised if the cloud gets very much of their actual scale at all.
>I can tell you just from digging into the publicly available videos let alone some of the not publicly available information that you are greatly underestimating the complexity and costs of Netflix’s architecture.
Why do they need all that complexity to host a bunch of (not even that many) videos and user profiling?
Why do they need all that complexity to host a bunch of (not even that many) videos and user profiling?
Search for Netflix’s reinvent talks on YouTube
Is there an argument in your comment?
For my argument not "all large" companies need to be dumb, as in the strawman above. It's enough to make more sense for the average company to do so. And there are tons of companies, large, SME and others, with big data to store, that have no ops or IT department as such, or just a couple of people.
They're not going to build datacenters of their own, distribute them globally, research into reliability and failover solutions, add some custom or adapt some third party stack, and so on. The salary for just an extra person could tower their AWS costs.
For my argument not "all large" companies need to be dumb, as in the strawman above. It's enough to make more sense for the average company to do so. And there are tons of companies, large, SME and others, with big data to store, that have no ops or IT department as such, or just a couple of people.
They're not going to build datacenters of their own, distribute them globally, research into reliability and failover solutions, add some custom or adapt some third party stack, and so on. The salary for just an extra person could tower their AWS costs.
When I did contracting in this space, my AWS based customers invariably needed far more of my time for similar sized setups than the non-cloud customers. The assumption that the non-cloud users will need to pay for any extra people is the opposite of my experience actually running both types of setup. I used to offer to cut their costs by moving them off AWS, but didn't push it very hard because I made far more money from the ones that insisted on using AWS. Some of them had good reasons. Many were just cargo-culting.
You don't need to build datacenters, or rack servers yourself, or even own servers, or even ever having seen your servers in order to have racks of dedicated bare metal colo racks of servers. Plenty of service-providers will rent you those services by the hour, and plenty of server providers will rent or lease-to-own hardware for you, so you get monthly bills.
I started working in this space in 1995, and while we've often had servers actually on prem, I've also managed racks worth of servers I've never seen in person, in data centres I've never seen, with the help of "remote hands" provided by colo provider employees I've never met.
You don't need to build datacenters, or rack servers yourself, or even own servers, or even ever having seen your servers in order to have racks of dedicated bare metal colo racks of servers. Plenty of service-providers will rent you those services by the hour, and plenty of server providers will rent or lease-to-own hardware for you, so you get monthly bills.
I started working in this space in 1995, and while we've often had servers actually on prem, I've also managed racks worth of servers I've never seen in person, in data centres I've never seen, with the help of "remote hands" provided by colo provider employees I've never met.
So did they have multiple redundant data centers worldwide?
They who? I've had dozens of customers, and multiple employers, and co-founded several companies. Most of them had servers in multiple redundant data centres worldwide. None owned any data centres because it rarely pays - owning data centres is a real estate play, not a tech play, which is why you also increasingly see real-estate management companies pick up data centre operators. Unless you're spending in the billions, it's rarely worth it when the markup is a tiny fraction of what you're paying for space and power.
Some w/rented servers. Some leased. Some owned. Some with staff on retainer just for us. Some just w/support contracts and remote hands. In some cases we rented office space for ops people in the colo facility, because again: They're real estate plays and the big ones have office suites available.
Some w/rented servers. Some leased. Some owned. Some with staff on retainer just for us. Some just w/support contracts and remote hands. In some cases we rented office space for ops people in the colo facility, because again: They're real estate plays and the big ones have office suites available.
I can’t quite grok from your original comment are you saying that every company who tried to duplicate what AWS does on prem themselves end up spending too much money on consultants etc or are you saying that companies that use AWS still spend too much money on consultants and they don’t have any cost savings?
Am saying that:
- most companies will do better to not try to replicate AWS (or the share of AWS functionality that they need)
- most companies aren't tech companies (not talking about the Netflixes and co of the world here) and would botch it, and at best get a costly + high maintenance bad copy
- most companies will do better to not try to replicate AWS (or the share of AWS functionality that they need)
- most companies aren't tech companies (not talking about the Netflixes and co of the world here) and would botch it, and at best get a costly + high maintenance bad copy
I've seen the size of discounts companies with magnitudes smaller discounts than Netflix has managed to get. I'm sure it's cost-effective for Netflix. That means nothing with respect to whether it will be cost-effective for someone else.
Exactly, not many internal solutions have 11 9s of durability across multiple AZs like S3.
Mostly because almost nobody needs it, and even fewer need it in that form. And if you need that, then even if AWS can deliver it, you need more than one supplier.
Because dealing with that much data spread across suppliers is not going to introduce complexity and latency.
Of course it will introduce complexity and latency, but if durability is so important that you need 11 9's, then you don't have a realistic choice unless all you care about is the appearance of it.
Put another way: I was around when WorldCom went bankrupt and their datacentre in half the building we were also in was locked down by administrators. Had a server fail? Tough. You wouldn't get in. They were not Amazon of today sized, but they had $20bn/year revenue.
That's one of many failure modes that can take you out if you rely on one provider. I'm uneasy about relying on one provider for far lower durability projects - I usually ensure we at least have backups somewhere else.
But if you care enough that you pick AWS due to a need for 11 nines, and you don't include the long list of failure scenarios that include anything from a bankruptcy (yes, it's unlikely), to them kicking you off intentionally or by accident, or you failing to pay a bill, or any number of other scenarios, they might have 11 nines durability, but you don't have a guarantee you'll get access to it.
I'm not arguing against using AWS. It's expensive, but it has its uses. I use it at my current employer because our customers want it, and it's not a const-sensitive product. Picking it as part of a strategy to manage risks is entirely valid.
But it's certainly not going to save you from complexity unless you cut corners because they get you to ignore risks.
Put another way: I was around when WorldCom went bankrupt and their datacentre in half the building we were also in was locked down by administrators. Had a server fail? Tough. You wouldn't get in. They were not Amazon of today sized, but they had $20bn/year revenue.
That's one of many failure modes that can take you out if you rely on one provider. I'm uneasy about relying on one provider for far lower durability projects - I usually ensure we at least have backups somewhere else.
But if you care enough that you pick AWS due to a need for 11 nines, and you don't include the long list of failure scenarios that include anything from a bankruptcy (yes, it's unlikely), to them kicking you off intentionally or by accident, or you failing to pay a bill, or any number of other scenarios, they might have 11 nines durability, but you don't have a guarantee you'll get access to it.
I'm not arguing against using AWS. It's expensive, but it has its uses. I use it at my current employer because our customers want it, and it's not a const-sensitive product. Picking it as part of a strategy to manage risks is entirely valid.
But it's certainly not going to save you from complexity unless you cut corners because they get you to ignore risks.
Well seeing that if AWS went bankrupt and couldn’t afford their servers, I assure you that the federal government would step in seeing how many government services are hosted both in the commercial, GovCloud and “secret” (an official type) regions not to mention how many financial companies are hosted there.
And every cloud provider has their own set of proprietary services that you are going to need to both design around and have staff trained for and constantly be testing for.
No “running K8s and open source software” is not the answer. You still need to integrate with thier IAM solution, how they manage their VMs other supporting infrastructure and often if you are large enough, you will have redundant physical connections to their data centers.
Also “using Terraform” is not the answer. A TF configuration is not portable at all across vendors.
And every cloud provider has their own set of proprietary services that you are going to need to both design around and have staff trained for and constantly be testing for.
No “running K8s and open source software” is not the answer. You still need to integrate with thier IAM solution, how they manage their VMs other supporting infrastructure and often if you are large enough, you will have redundant physical connections to their data centers.
Also “using Terraform” is not the answer. A TF configuration is not portable at all across vendors.
> Well seeing that if AWS went bankrupt and couldn’t afford their servers, I assure you that the federal government would step in seeing how many government services are hosted both in the commercial, GovCloud and secret regions not to mention how many financial companies are hosted there.
Maybe. Worldcom was eventually bought out too. That does not mean you can assume that there will be no disruption to services, or no loss. And that addressed one of many potential risks. Unless you can guarantee that there are no such risks whatsoever, the point I made stands.
That point was simply that if you rely on a third party's durability level, then whatever that level is, the durability of the data from your point of view is lower, because there will always be a risk that your access to the data is removed either permanently or long enough for the eventual recovery to be irrelevant.
And so if you actually need and care about that durability level, you'll need multiple providers. Which you probably should at far lower reliability levels anyway.
I did not suggest it'd be simpler to use another cloud provider, so I have no idea why you're bringing that up as it's entirely irrelevant to the argument I made.
Maybe. Worldcom was eventually bought out too. That does not mean you can assume that there will be no disruption to services, or no loss. And that addressed one of many potential risks. Unless you can guarantee that there are no such risks whatsoever, the point I made stands.
That point was simply that if you rely on a third party's durability level, then whatever that level is, the durability of the data from your point of view is lower, because there will always be a risk that your access to the data is removed either permanently or long enough for the eventual recovery to be irrelevant.
And so if you actually need and care about that durability level, you'll need multiple providers. Which you probably should at far lower reliability levels anyway.
I did not suggest it'd be simpler to use another cloud provider, so I have no idea why you're bringing that up as it's entirely irrelevant to the argument I made.
“We’re highly available within a single data center”
- CIO of a bank that I was working with
For storage in particular this gets dicey. Is there an off site backup? What data loss is acceptable for that recovery? What availability requirements do you have?
Tales of faulty drives destroying companies are everywhere.
- CIO of a bank that I was working with
For storage in particular this gets dicey. Is there an off site backup? What data loss is acceptable for that recovery? What availability requirements do you have?
Tales of faulty drives destroying companies are everywhere.
After the '96 OKC bombing, banks came to a whole new appreciation of redundancy. I was only a few years out of high school working freelance gigs, but one of them was following the progress of a specific bank implementing primary/secondary/tertiary sites and how long the secondary becomes the primary and the tertiary is promoted to secondary. Obviously, this was pre-cloud, so it was even more of a herculean task. It was also the first time I saw a large tape library with robot changer, so it sticks out in the memory banks.
errr, '95 OKC bombing. <facepalm> I'd love to call it a typo, but I definitely misremembered the date.
Most likely reason is now the data transfer truck is more likely to have people remove their data than add it to the AWS cloud.
Or global warming. Yamaha is also killing their snowmobile.
https://snowgoer.com/latest-news/yamaha-will-exit-the-snowmo...
https://snowgoer.com/latest-news/yamaha-will-exit-the-snowmo...
Bummer, I always enjoyed joking about this
“Tell your data to get Trucked”
“Tell your data to get Trucked”
Why "Snowmobile"? How does that relate to a truck full of data?
They first introduced a product called Snowball which is a big briefcase of hard drives that would ship you, you transfer up to a petabyte of data and send back. This was the extension of that so they went with another snow related name. Also their data archival service called Glacier which all imply "cold storage."
It's a great marketing gimmick. Everyone fell for it
Yeah, it's shame they didn't spruce it up a bit. e.g. some winter theming on the exterior, allowing for some commemorative swag at conferences like they had for the warehouse robots or the delivery drones (also largely gimmicks).
Makes sense. Storage tech has moved significantly since
The snowmobile was for data transfer, not data storage. The idea was that you upload your data to the truck on-site, then they drive it to the backup and download it again, faster than you could transfer it over the wire
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981
Yes, but I suspect the parent meant that storage density has significantly increased, so you now don't need an entire truck's worth of it. The other "AWS Snow" options may now suffice? You gotta wonder how many companies actually used the truck...
Eyeballing this chart: https://en.wikipedia.org/wiki/File:Hard_drive_capacity_over_...
It looks like over the past 8 or so years you're looking at a maybe a tripling in capacity. Cut the size of the truck down by a third, accounting for the fact the truck couldn't have been stuffed with every cubic inch being hard drive due to the need to walk in it, weight limits, cooling, wiring, etc., and I think it's not hard to see that even if it did make sense at the time, it's not hard to run the numbers and build some 4-6 U storage units and ship those instead. Even if you ship 4 of those it's cheaper than a truck by quite a bit.
Also, if you're willing to spend the dough, flash density can be pushed even higher, I think. You might need some custom hardware to manage power to the drives, so only the drives the customer is filling right now are powered (heat management is going to be a big deal with flash), but that's not impossibly difficult.
You can even drain an entire data center by cycling the same 3 or 4ish units around. Fill one up, send it in, Amazon loads it up, send the same unit back. Even if it takes 10 loads you don't necessarily need 10 hunks of hardware.
Personally I have the sneaking suspicion that it was always a marketing stunt and never actually made monetary sense. It was just a grand gesture to reassure people that they could indeed move to the cloud, which was a large concern for a lot of large customers. In that sense it probably paid for itself. And once you do have it in hand, maybe the marginal costs are still attractive and you go ahead and use it sometimes. But I'd personally bet the "round robin of 4-6U storage racks" would always have been more economical regardless of hard drive densities.
It looks like over the past 8 or so years you're looking at a maybe a tripling in capacity. Cut the size of the truck down by a third, accounting for the fact the truck couldn't have been stuffed with every cubic inch being hard drive due to the need to walk in it, weight limits, cooling, wiring, etc., and I think it's not hard to see that even if it did make sense at the time, it's not hard to run the numbers and build some 4-6 U storage units and ship those instead. Even if you ship 4 of those it's cheaper than a truck by quite a bit.
Also, if you're willing to spend the dough, flash density can be pushed even higher, I think. You might need some custom hardware to manage power to the drives, so only the drives the customer is filling right now are powered (heat management is going to be a big deal with flash), but that's not impossibly difficult.
You can even drain an entire data center by cycling the same 3 or 4ish units around. Fill one up, send it in, Amazon loads it up, send the same unit back. Even if it takes 10 loads you don't necessarily need 10 hunks of hardware.
Personally I have the sneaking suspicion that it was always a marketing stunt and never actually made monetary sense. It was just a grand gesture to reassure people that they could indeed move to the cloud, which was a large concern for a lot of large customers. In that sense it probably paid for itself. And once you do have it in hand, maybe the marginal costs are still attractive and you go ahead and use it sometimes. But I'd personally bet the "round robin of 4-6U storage racks" would always have been more economical regardless of hard drive densities.
Anyone have an updated estimate for how much storage could be fit in a shipping container and its bandwidth equivalent?
[https://news.ycombinator.com/item?id=1437634#:~:text=Conserv....] says 60PB, which for 20 TB drives would be approx 3000.
A petabyte is still a lot of data!
In my experience the bigger issue is that much large data sets are rarely truly static, and inventorying/tracking/validating is a massive undertaking all on it’s own.
[https://news.ycombinator.com/item?id=1437634#:~:text=Conserv....] says 60PB, which for 20 TB drives would be approx 3000.
A petabyte is still a lot of data!
In my experience the bigger issue is that much large data sets are rarely truly static, and inventorying/tracking/validating is a massive undertaking all on it’s own.
Obligatory XKCD: https://what-if.xkcd.com/31/
As a side note since XKXD leaves this out and having gotten involved in implementing systems using physical storage for transport in the past, physical copying to/from media is a major issue in performance and capital costs.
Most large physical hard drives max out around 300-400MB/s transfer rate, so it typically takes several days to fill a single drive. And you can’t ship anything until all the media is written.
So either you have a ton of online, parallel drives (which dramatically increases capex, and decreases available storage due to physical space, power, and cooling issues) or you go really slow.
Even having 300 parallel full speed drive interfaces for instance, it would take approximately a year to fill those 3000 drives with data, and approx. another year to read all the data off. Which dramatically changes all the calculations.
Most large physical hard drives max out around 300-400MB/s transfer rate, so it typically takes several days to fill a single drive. And you can’t ship anything until all the media is written.
So either you have a ton of online, parallel drives (which dramatically increases capex, and decreases available storage due to physical space, power, and cooling issues) or you go really slow.
Even having 300 parallel full speed drive interfaces for instance, it would take approximately a year to fill those 3000 drives with data, and approx. another year to read all the data off. Which dramatically changes all the calculations.
The number of companies with that problem is going to naturally fall over time. Companies that are already using the cloud don't need Snowmobile to move their data.
Illustrative point from the article: "A spokesperson for satellite operator Maxar said the company used Snowmobile once in 2017 to move more than 100 petabytes to AWS from its own servers. Snowmobile was an “ideal solution for that moment in time,” the spokesperson said."
You're not going to get many repeat customers for that service!