AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage(twitter.com)
twitter.com
AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
https://twitter.com/adrianco/status/1560854827810361345
206 comments
I doubt there’s a single byte stored to disk that is not compressed and encrypted at AWS. It’s transparent to the customer.
I don't think it's so clear cut. They have to pay to compress it. If the data the customer stores is short lived it may not be worth it to them. They don't know if the customer already compressed it so they might be wasting their CPU. They also have to pay to decompress it on every access. They allow you to slice an arbitrary byte range out of an object which is technically harder to implement on a compressed file. They charge by the GB and are not exactly super cheap so if the customer wants to store a big fat file of easily compressible zeros then whatever, they got their money.
It might make more sense on their "deep archive" product maybe where the customer has to commit to a minimum storage retention and also pay a retrieval charge which scales with the amount of data recovered (hence paying for the CPU to decompress).
It might make more sense on their "deep archive" product maybe where the customer has to commit to a minimum storage retention and also pay a retrieval charge which scales with the amount of data recovered (hence paying for the CPU to decompress).
I don’t work at AWS, but storage at scale is a funny beast, usually you’re constrained by IOPS, and if anything you have a surplus of CPU.
If you can stuff more bits in an IO operation, you’re winning.
If you can stuff more bits in an IO operation, you’re winning.
Moreover zstd quite unusual '--adapt' parameter enables it to "dynamically adapt compression level to perceived I/O conditions". Works for me (albeit the manpage states that "it can remain stuck at low speed when combined with multiple worker threads").
Too bad the flag doesn't come with detection for this environmental condition and then coordinate accordingly across processes.
Is there a paper on how it "perceives" the I/O conditions?
I'd guess by using backpressure.
Modify the compression level to try and keep the output buffer at 60% full.
Modify the compression level to try and keep the output buffer at 60% full.
It reacts to the input buffer state (in bad I/O conditions it starves).
On Linux PSI ( /proc/pressure/io ) probably provides more accurate information (the code already uses /proc/cpuinfo ).
Detail: in the fileio.c module there are lines such as:
if (oldIPos == inBuff.pos) inputBlocked++; /* input buffer is full and can't take any more : input speed is faster than consumption rate /
if ( (inputBlocked > inputPresented / 8) / input is waiting often, because input buffers is full : compression or output too slow */
This impacts a 'speedChange' variable.
Its potential values (an enum) are 'noChange', 'slower', and 'faster'. They are processed rather simply: if (speedChange == slower) { ((...)) compressionLevel ++;
if (speedChange == faster) { ((...)) compressionLevel --;
On Linux PSI ( /proc/pressure/io ) probably provides more accurate information (the code already uses /proc/cpuinfo ).
Detail: in the fileio.c module there are lines such as:
if (oldIPos == inBuff.pos) inputBlocked++; /* input buffer is full and can't take any more : input speed is faster than consumption rate /
if ( (inputBlocked > inputPresented / 8) / input is waiting often, because input buffers is full : compression or output too slow */
This impacts a 'speedChange' variable.
Its potential values (an enum) are 'noChange', 'slower', and 'faster'. They are processed rather simply: if (speedChange == slower) { ((...)) compressionLevel ++;
if (speedChange == faster) { ((...)) compressionLevel --;
[deleted]
I think it is a clear cut, mostly because I do not think that compression compromises any of those features, all while making the user experience better.
For any storage system like this you usually have a few bottlenecks. IO and Network are the obvious ones, followed by tiering (cache, fast io, slow io, ...) and at the very end CPU.
Now let's say network is your bottleneck. If you can send the data to the client in a compressed for then you get the compression ratio as additional bandwidth. And the user would get the data quicker! So compression to the network is a clear win.
But the common bottleneck is often IO, a high end SSDs with 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit link. That's without any redundancy over other overhead.
Again compression to the storage layer would decrease the total amount of IOs, thus making sure a customer gets data quicker.
Ok, let's say both are not the issue. The fastest compression algorithms compete with memcopy. So if you need just one copy of your data you might have been faster by compressing it.
Especially fast compression algorithms (zstd, lz4, snappy, lzo, ...) are worth the CPU cost with virtually no downsides. The problem is finding the right sweet spot that reduces the current bottleneck without creating a CPU bottleneck, but zstd offers the greatest flexibility there, too.
Oh for range requests.... Those large objects are likely split anyway, for easier error recovery (imagine 100MB into a 1GB transfer you notice that the file data was corrupted - not good). Once you work on blocks it's easy to do somewhat efficient range requests again.
For any storage system like this you usually have a few bottlenecks. IO and Network are the obvious ones, followed by tiering (cache, fast io, slow io, ...) and at the very end CPU.
Now let's say network is your bottleneck. If you can send the data to the client in a compressed for then you get the compression ratio as additional bandwidth. And the user would get the data quicker! So compression to the network is a clear win.
But the common bottleneck is often IO, a high end SSDs with 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit link. That's without any redundancy over other overhead.
Again compression to the storage layer would decrease the total amount of IOs, thus making sure a customer gets data quicker.
Ok, let's say both are not the issue. The fastest compression algorithms compete with memcopy. So if you need just one copy of your data you might have been faster by compressing it.
Especially fast compression algorithms (zstd, lz4, snappy, lzo, ...) are worth the CPU cost with virtually no downsides. The problem is finding the right sweet spot that reduces the current bottleneck without creating a CPU bottleneck, but zstd offers the greatest flexibility there, too.
Oh for range requests.... Those large objects are likely split anyway, for easier error recovery (imagine 100MB into a 1GB transfer you notice that the file data was corrupted - not good). Once you work on blocks it's easy to do somewhat efficient range requests again.
How clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
What I've personally implemented is trial compression with heuristics (you eagerly compress chunks, and if enough chunks don't compress, stop trying). It does require low level input/output control and per-chunk/block compression.
That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.
That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.
This is not how such storage systems work. If the whole storage system is for you and you only store compressed video files on it then perhaps.
But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers.
All the benefits hold, even for you with non compressible data, if there is an overall benefit.
Hitting a bottleneck less often then this will reduce your tail latencies, too.
It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.
But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers.
All the benefits hold, even for you with non compressible data, if there is an overall benefit.
Hitting a bottleneck less often then this will reduce your tail latencies, too.
It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.
No, most compressed storage systems do not waste any CPU on data that is not compressible.
Having a big CPU (which you need to get many PCIe lanes) and then not using it is waste.
It would be waste to _not_ use the CPU time.
Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep)
I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places.
Do not waste the things you pay for!
It would be waste to _not_ use the CPU time.
Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep)
I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places.
Do not waste the things you pay for!
Video files have already entropy coding applied to them and thus any compression gains with reapplying a generic entropy coding like zstd are unlikely:
https://en.wikipedia.org/wiki/Entropy_coding
https://en.wikipedia.org/wiki/Entropy_coding
That's exactly what the poster is saying.
They could be using hardware compression which can be orders of magnitude faster than doing it on the CPU.
Hardware compression is sadly not widely available, I think the only consumer product I know with it is the PlayStation 5.
The mainframes from IBM have had hardware zlib since Z14 iirc and in my small tests it is very fast compared to the CPU implementation
Hardware compression is sadly not widely available, I think the only consumer product I know with it is the PlayStation 5.
The mainframes from IBM have had hardware zlib since Z14 iirc and in my small tests it is very fast compared to the CPU implementation
Hardware compression IS available in Graviton 2:
"1Tbit/s of compression accelerators
• 2xlarge and larger instances will have a compression device
• DPDK and Linux kernel drivers will be available ahead of GA
• Data compression at up to 15GB/s and decompression at up
to 11GB/"
I wonder if this could be done on a FPGA..
FPGA for development, AWS can afford to order custom ICs.
With enough gates you can do anything on an FPGA
How much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.
Probably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high).
I would expect this to become part of newer generations of CPUs once it becomes popular.
I would expect this to become part of newer generations of CPUs once it becomes popular.
Intel sells a gzip PCIe card called QAT. Not many seem to be sold.
I'd never heard of this! The full name is QuickAssist and it does encryption too. They advertise 100Gb/s symmetric crypto, 70Gb/s compression (or roughly 100x faster than ZStandard on a single CPU). Seems to retail for about $650 for a card.
It seems like QuickAssist can also be used for Transcoding. Plex e.g. supports it.
My Intel i9 11900k seems to support it or has it built-in.
My Intel i9 11900k seems to support it or has it built-in.
QAT is actually likely to end up inside new server CPUs from Intel - at least according to the advertising material. Also, it is in their new SmartNICs. At least somebody is using it.
It has been done before, if you offload gzip you turn your PCI bus into a choke point. Most of the time you do better keeping it on the main processor.
I think a lot if datacenter SSDs already come with in-drive hardware compression these days, since it not only increases speed but also longevity. So it would actually save money anyways.
They do not. It would be difficult to plan correctly if your free disk space is… variable.
Example:
You have an existing 40 gigabyte file
It happened to compress well
You delete it and your free disk space goes up by 4 gigabytes.
You then write a new 40 gigabyte file that doesn’t compress well
Replacing an existing file of the same size just ate an extra 36 gigabytes.
How would you plan around that? SSDs should store the bytes given and don’t play fancy games.
Example:
You have an existing 40 gigabyte file
It happened to compress well
You delete it and your free disk space goes up by 4 gigabytes.
You then write a new 40 gigabyte file that doesn’t compress well
Replacing an existing file of the same size just ate an extra 36 gigabytes.
How would you plan around that? SSDs should store the bytes given and don’t play fancy games.
It's not great, but it's also not unheard of. Tape capacities are often quoted at double the actual storage space, with fine print that says "assuming 50% compression". Also, if compression makes IOPs faster or reduces wear on SSDs, people might not complain so much.
SSDs could still do it for speed and to write less pages, making the drive last longer, and simply report the uncompressed space as used. They already do all sorts of tricks on pages such as moving them logically, having more internally than they report to use as pages wear out, and so on.
Given that, it's good sense to compress if at all possible simply to make the drive live longer.
And guess what - I just googled, tons of hits, and this has been done for a long time :)
So it makes sense, is done, and is important for modern SSD behavior.
Given that, it's good sense to compress if at all possible simply to make the drive live longer.
And guess what - I just googled, tons of hits, and this has been done for a long time :)
So it makes sense, is done, and is important for modern SSD behavior.
None of this works if the data is encrypted.
I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it.
Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.
I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it.
Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.
> I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough
There's very, very little benefit in encrypting data at a filesystem level in a datacenter if you think about it.
There's very, very little benefit in encrypting data at a filesystem level in a datacenter if you think about it.
I explained why I think it does provide a benefit even in a datacenter:
"Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal."
Would you resell usable drives that you no longer want to use (e.g. because they're too small or your needs changed more towards SSDs) if unencrypted data was written to them at some point?
How much effort would you put into making sure that no broken drive that can't be wiped leaves the datacenter without shredding?
Any process you put in place will have gaps - e.g. through human error or malicious acts - and this provides pretty solid protection against that.
"Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal."
Would you resell usable drives that you no longer want to use (e.g. because they're too small or your needs changed more towards SSDs) if unencrypted data was written to them at some point?
How much effort would you put into making sure that no broken drive that can't be wiped leaves the datacenter without shredding?
Any process you put in place will have gaps - e.g. through human error or malicious acts - and this provides pretty solid protection against that.
Unfortunately that's going to really depend. For example, if your threat model is "hard drive gets stolen" there's no point. If your threat model is "attacker can access my database" encrypting the data at the DB level does make sense. But it obviously breaks compression.
And unfortunately compression and encryption are seemingly at odds fundamentally :c
And unfortunately compression and encryption are seemingly at odds fundamentally :c
Plenty of SSDs implement strong encryption, and there's an entire standard API for it. Why waste cpu on tasks your hardware already does?
Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't.
If I recall, it's a FIPS requirement for data at rest now.
Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't.
If I recall, it's a FIPS requirement for data at rest now.
> Why waste cpu on tasks your hardware already does?
Because the storage hardware might be loyal to someone who is not the owner of the data.
Because the storage hardware might be loyal to someone who is not the owner of the data.
> Why waste cpu on tasks your hardware already does?
Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason.
https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)
Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason.
https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)
Yes they do. Intel writes this for example (not all drives use it, but some definitely do):
>Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial.
https://www.intel.com/content/www/us/en/support/articles/000...
>Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial.
https://www.intel.com/content/www/us/en/support/articles/000...
Note that these are pre-2017 consumer SSDs. I think SSD compression fell out of favor due to the rise of FDE.
FDE can be done on the SSD after compression, and often is.
Adding life to SSDs is a terribly useful feature
Adding life to SSDs is a terribly useful feature
Isn't it solved by adding an abstraction layer between the storage and the filesystem, the way "vdo" ( https://github.com/dm-vdo/vdo ) does it?
What do you mean? Even without compression, you have half the problem. Obviously, at scale, it’s all just statistics and planning on trends instead of individual files.
how is this any different than e.g. fs level compression with zstd on btrfs? seems enough people find that useful.
>. They charge by the GB and are not exactly super cheap so if the customer wants to store a big fat file of easily compressible zeros then whatever, they got their money.
Maybe they charge at uncompressed rate but store it at compressed? Then they got even more money!
Maybe they charge at uncompressed rate but store it at compressed? Then they got even more money!
This was true a few years back, nowadays it’s cheaper and faster to compress the data at rest as the bottleneck is frequently IO and storage space. Both, in terms of capacity and cost.
This was true a few years back
Only temporarily with SSDs. With spinning rust, it also often paid off to compress data. We'd store large treebanks compressed, because decompression was much faster than disk reads.
Only temporarily with SSDs. With spinning rust, it also often paid off to compress data. We'd store large treebanks compressed, because decompression was much faster than disk reads.
[deleted]
It would still produce some CPU overhead, and thus some energy usage.
Presumably it's the tradeoff of CPU overhead versus disk and bandwidth (larger files take longer to copy into memory, which is also energy usage. And more bandwidth to shunt around Amazon's own network)
There's also a latency component.
Since CPUs are fast enough to deflate in real-time now, your bottleneck for a read is your storage/network.
Reducing the bytes read from storage improves the IO latency.
Since CPUs are fast enough to deflate in real-time now, your bottleneck for a read is your storage/network.
Reducing the bytes read from storage improves the IO latency.
[deleted]
> They allow you to slice an arbitrary byte range out of an object which is technically harder to implement on a compressed file.
this is pretty easy, you flush the compression buffer every megabyte or so and maintain an index
maybe 50 lines of code
this is pretty easy, you flush the compression buffer every megabyte or so and maintain an index
maybe 50 lines of code
Sure, but now you've added an extra layer of indirection which can have a significant impact on performance
It doesn't really have to impact performance. The index is generated easily as a side-effect of compression. And the index is only needed if you need to seek.
I implemented this as part of the MinIO server. See "Seeking Compressed Files" here: https://blog.min.io/transparent-data-compression/
We choose a compressor without literal compression for a faster baseline, but the concept remains the same.
I implemented this as part of the MinIO server. See "Seeking Compressed Files" here: https://blog.min.io/transparent-data-compression/
We choose a compressor without literal compression for a faster baseline, but the concept remains the same.
But if you do need to seek, which is really common in data warehouse workloads for example, unless you keep the index in ram you have to do an extra IO on every seek to read the index
there's always going to be some metadata for the file that needs to be looked up before you can start seeking (ACLs, sector/extent/cluster location, etc)
the index goes in there, no extra seek needed
the index goes in there, no extra seek needed
yeah that isn't free either, it adds significant bloat to your metadata. with most enterprise customers encrypting and/or compressing data before putting it into s3, it doesn't seem like there would be much benefit. s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
> yeah that isn't free either, it adds significant bloat to your metadata
yeah, 4 bytes for every megabyte
> s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
yeah, I'm sure you're right and Amazon have absolutely no idea what they're doing and like to spend unnecessary CPU cycles doing pointless work and add "significant bloat" to their metadata
... or, you're wrong (like in every previous comment in this chain)
yeah, 4 bytes for every megabyte
> s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
yeah, I'm sure you're right and Amazon have absolutely no idea what they're doing and like to spend unnecessary CPU cycles doing pointless work and add "significant bloat" to their metadata
... or, you're wrong (like in every previous comment in this chain)
https://www.reddit.com/r/programming/comments/wtd61q/aws_swi...
this tweet is not talking about compressing customer data in s3, i seriously doubt that aws compresses customer data in s3 for all the reasons i've already listed. i am right and amazon does know what they're doing, which is why they don't compress customer data in s3.
4 bytes per megabyte becomes significant at scale when you have to keep it in ram, which you have to do if you want to avoid the extra IO.
this tweet is not talking about compressing customer data in s3, i seriously doubt that aws compresses customer data in s3 for all the reasons i've already listed. i am right and amazon does know what they're doing, which is why they don't compress customer data in s3.
4 bytes per megabyte becomes significant at scale when you have to keep it in ram, which you have to do if you want to avoid the extra IO.
You only need a single part to calculate a specific offset, assuming you have part sizes stored in metadata already (a good idea).
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.
ah yes, "authoritative" comments from random reddit accounts
and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't
and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't
> They charge by the GB and are not exactly super cheap so if the customer wants to store a big fat file of easily compressible zeros then whatever, they got their money.
That's not a good argument: they could lower their costs with compression, still charge the same, and make more profit.
That's not a good argument: they could lower their costs with compression, still charge the same, and make more profit.
Why do they have to either compress it all or not. They must be smart like, have the files split in pieces (just like some network file systems/backups do) and if those blocks are untouched for a while, compress what's compressible and leave them as is during active uses.
I think the point of the different storage tiers of AWS S3 is to get customers to classify their own data, then AWS can pick the right mix of hardware, software, and compute that satisfies AWS’s requirements for availability and COGS.
If the difference between standard S3 and S3 Glacier was just slower disk, then rate limiting the customer would suffice.
But if there’s a significant amount of compute thrown at data de-duplication, compression, and indexing, then it starts to clarify why there’s a pricing penalty for using Glacier with the same access patterns as one would use on standard storage.
If the difference between standard S3 and S3 Glacier was just slower disk, then rate limiting the customer would suffice.
But if there’s a significant amount of compute thrown at data de-duplication, compression, and indexing, then it starts to clarify why there’s a pricing penalty for using Glacier with the same access patterns as one would use on standard storage.
> They charge by the GB and are not exactly super cheap so if the customer wants to store a big fat file of easily compressible zeros then whatever, they got their money.
And what if they can charge you the uncompressed size and only actually store much smaller compressed files behind the scene. That seems like having your cake and eat it too.
And what if they can charge you the uncompressed size and only actually store much smaller compressed files behind the scene. That seems like having your cake and eat it too.
I think an FPGA can probably compress quite a bit faster than a general purpose CPU, and an ASIC on a card which is network on one side and storage on the other could compress at line speed easily.
Amazon has millions of idle cpus available 24 hours a day (they can use all the idle time for all customer instances for whatever they want)
That doesn't make it completely free. They still have opportunity costs.
Idle CPU's also use less power than ones running full tilt.
+1 if you are storing objects uncompressed, I'd be amazed if AWS doesn't compress them and charge you for the full space anyway
If this true, there is possibly a side channel one could run against object storage to determine if someone else in the content-addressable-store has the same files.
Like when it was easy to file share on dropbox by having the correct hashes. A GUID could summon a 1GB file.
Like when it was easy to file share on dropbox by having the correct hashes. A GUID could summon a 1GB file.
Compression and content-addressing are two separate things.
Content addressing across accounts on private, AWS encrypted S3 buckets would run counter to their claims.
Content addressing across accounts on private, AWS encrypted S3 buckets would run counter to their claims.
A couple of comments across the thread have made similar points, but if I were implementing this, the "client metadata" like the incoming sha256 etc would be implemented a layer higher than the actual byte storage, so the byte storage could be compressed without any impact on that sort of thing.
That assumes cross-tenant compression.
It's not quite that simple. If you have customer data that's already encrypted, then compression won't do much because it looks random. But of course by the time you get to your infrastructure layers, that'll be the case (or you really messed up your security story!). Which means you'd have to compress right at the edge. They might be doing that (which would basically mean it's the customer compressing it before they encrypt it with their keys because AWS has no business seeing the clear text), but then you get to compress each item separately, which might not be very effective for small values.
tl;dr:There's a real efficiency/security/insider risk trade-off here.
Edit: I should disclose that I work for a competitor. Don't intend any astroturfing.
tl;dr:There's a real efficiency/security/insider risk trade-off here.
Edit: I should disclose that I work for a competitor. Don't intend any astroturfing.
I agree with you, there could be scenarios where customers supply their own keys and compress the data on their own. My original statement is still true though, the data at rest ends up being compressed and encrypted.
That said, of course, customers can upload encrypted blobs of uncompressed data. But I’d call it an exception that proves the rule. Here service simplicity should win and those blobs may end up recompressed.
That said, of course, customers can upload encrypted blobs of uncompressed data. But I’d call it an exception that proves the rule. Here service simplicity should win and those blobs may end up recompressed.
How would that work for something like S3 range requests? Rather than reading an entire object sequentially (which would work fine with transparent compression) you can also ask to read an arbitrary byte range (give me bytes 1,000,000,000-1,000,001,000 from the original file). I guess you could maybe store the compressed file in chunks with metadata about the original byte range inside each chunk.
For MinIO (an S3 compatible server), we add an index for each part, which contains uncompressed -> compressed offset pairs.
Since we already used a Snappy-derived method, each 1MB block is stored without backreferences. With this we only have to decode at most 1MB-1 extra bytes to respond with a specific range offset.
Since we already used a Snappy-derived method, each 1MB block is stored without backreferences. With this we only have to decode at most 1MB-1 extra bytes to respond with a specific range offset.
Generally with filesystem-level compression you don't compress an entire multi-GB file: you compress segments of maybe a few 100k. This gives you a very slightly worse compression ratio but allows random seeks to still be efficient.
[deleted]
[deleted]
I'm sure it's encrypted, but I doubt that they compress everything. Images and video tend not too compress well since they've typically already been aggressively compressed with specialized algorithms. It would just be throwing CPU cycles away.
They do compress for most log delivery types like load balancers, cdn, cloudtrail, etc. and it makes a huge difference over raw logs - compression ratios are in the 90s. One AWS specific trick is individual log files below some size threshold are passed through raw, so you end up with a mix of compressed and uncompressed files in S3, and the only reliable way to distinguish between them is to receive them and look for a zlib header - you can’t depend on the file name or any other metadata to tell you ahead of time if an individual file is compressed or not. (I think Cloudtrail does set metadata correctly for uncompressed files, but other types do not, best to spend the time developing a abstraction that deals with both)
Amazon is AWS's biggest customer, but I've never been told to compress things with zstd while working at AWS. And services still compress files as gzip before storing data in S3.
Maybe data in S3 is compressed? I don't know the internals of S3.
Maybe data in S3 is compressed? I don't know the internals of S3.
> Amazon is AWS's biggest customer, but I've never been told to compress things with zstd while working at AWS.
You may have switched transparently to using it (e.g. for your application log files), via internal Amazon tooling (that many AWS services use).
However, this tweet wasn't about AWS customer's data, but AWS's own data.
> Maybe data in S3 is compressed? I don't know the internals of S3.
Almost every other AWS service uses S3 for storing something.
Think of EBS snapshots, DynamoDB backups, RDS backups.
And that's not even the service-internal data.
You may have switched transparently to using it (e.g. for your application log files), via internal Amazon tooling (that many AWS services use).
However, this tweet wasn't about AWS customer's data, but AWS's own data.
> Maybe data in S3 is compressed? I don't know the internals of S3.
Almost every other AWS service uses S3 for storing something.
Think of EBS snapshots, DynamoDB backups, RDS backups.
And that's not even the service-internal data.
It’s also faster to decompress. So it would likely reduce net CPU use for read-mostly resources.
Everyone's interpreting this as AWS is zipping every customer's S3 data and improved storage of their data with ztsd.
That's now what Adrian said, and this is not an announcement - this was a "by-the-way" from a retired Amazonian.
AWS switched our own service's log storage (in S3) from gzip to ztsd a bunch of years ago -and reduced storage costs by 30%.
This is AWS's service data (which is still exabytes) and AWS realized the efficiency.
So all the comments of "where is my 30% discount" are offbase.
Source: I've worked on AWS for a bunch of years.
AWS switched our own service's log storage (in S3) from gzip to ztsd a bunch of years ago -and reduced storage costs by 30%.
This is AWS's service data (which is still exabytes) and AWS realized the efficiency.
So all the comments of "where is my 30% discount" are offbase.
Source: I've worked on AWS for a bunch of years.
It uses some black magic wizardy
>FSE, short for Finite State Entropy, is an entropy codec based on ANS. FSE encoding/decoding involves a state that is carried over between symbols, so decoding must be done in the opposite direction as encoding. Therefore, all FSE bitstreams are read from end to beginning. Note that the order of the bits in the stream is not reversed, we just read the elements in the reverse order they are written.
https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
>FSE, short for Finite State Entropy, is an entropy codec based on ANS. FSE encoding/decoding involves a state that is carried over between symbols, so decoding must be done in the opposite direction as encoding. Therefore, all FSE bitstreams are read from end to beginning. Note that the order of the bits in the stream is not reversed, we just read the elements in the reverse order they are written.
https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
For what it's worth, shameless plug, I've written a whole series on how FSE works: https://shargs.github.io/data-compression/2022/07/28/entropy...
It's not that crazy if you separate it into two parts.
A) Use arithmetic coding to encode symbols as fractions instead of using an integer number of bits for each symbol. https://en.wikipedia.org/wiki/Arithmetic_coding http://www.ws.binghamton.edu/fowler/fowler%20personal%20page...
B) Shuffle the bits around so that you can use faster math and lookup tables instead of directly operating on fractions.
A) Use arithmetic coding to encode symbols as fractions instead of using an integer number of bits for each symbol. https://en.wikipedia.org/wiki/Arithmetic_coding http://www.ws.binghamton.edu/fowler/fowler%20personal%20page...
B) Shuffle the bits around so that you can use faster math and lookup tables instead of directly operating on fractions.
I'm guessing he's NOT talking about S3's user data.
He's talking about runtime logs of the S3 service, or possibly logs from Amazon's services that are written to S3.
Amazon has ~100K services running on millions of EC2 instances. (Generalizing) each service writes logs to the local machine, and then later compresses and transmits them to a set of systems that are backed by S3 for monitoring / further bulk processing etc.
Some back of the envelope math on this:
For 1 exabyte to be the number, that equates to < 1TB average logs kept per host. Depending on what the service is doing log retention is 3mo / 1y / 10y. Using those numbers we're talking per service average 12-460 MB/hour.
He's talking about runtime logs of the S3 service, or possibly logs from Amazon's services that are written to S3.
Amazon has ~100K services running on millions of EC2 instances. (Generalizing) each service writes logs to the local machine, and then later compresses and transmits them to a set of systems that are backed by S3 for monitoring / further bulk processing etc.
Some back of the envelope math on this:
For 1 exabyte to be the number, that equates to < 1TB average logs kept per host. Depending on what the service is doing log retention is 3mo / 1y / 10y. Using those numbers we're talking per service average 12-460 MB/hour.
Yep, someone from aws clarified this on r/programming:
https://old.reddit.com/r/programming/comments/wtd61q/aws_swi...
https://old.reddit.com/r/programming/comments/wtd61q/aws_swi...
I want to know why a company as big as Amazon was still using gzip in 2022....
Sure - it's a handy tool. But at scale surely someone had enough time to research better options? And at Amazon scale, it probably even pays to hire a team to write a custom compression algorithm perfectly tuned to your compute and storage?
Sure - it's a handy tool. But at scale surely someone had enough time to research better options? And at Amazon scale, it probably even pays to hire a team to write a custom compression algorithm perfectly tuned to your compute and storage?
If I were Amazon, researching whether zstd can be used reliably could take quite some time. It would require a thorough code review, inspection of the assembly produced by compilers, and a test set of a few terabytes.
More importantly, it would require researching whether something even better might be just around the corner. I don’t think Amazon wants to switch compression every few months.
More importantly, it would require researching whether something even better might be just around the corner. I don’t think Amazon wants to switch compression every few months.
Huh. If an IC had a compelling argument for cost savings like this, I’d recommend they pursue the idea as a 10% style project. First verify the basics: licensing, health of the community, etc. Then build a standalone pipeline using the new protocol. Process data, collect benchmarks and metrics. Present your findings to the team and manager. If the manager thinks it’s still viable, they go up the chain and involve principals, architects and so on.
In summary, lower the bar for trying new things. This is how we innovate.
In summary, lower the bar for trying new things. This is how we innovate.
That's literally what the person above you said. On Amazon's scale the process will take time.
[deleted]
He did not say in the tweet zstd was implemented at AWS in 2022, could have been implemented years ago
When I was there last over a year ago, there was a big push to switch logs stored in the internal log service from gzip to zstd. But gzip is extremely well integrated into essentially everything, that it's a little annoying to switch. For example, the base OS images didn't even have zstd installed. It's still worth it for the benefits (at least once all the internal tools started supporting it), but it's not without some downsides. Which is maybe why it took so long to happen in earnest.
Gzip was written extremely well. I have been trying to get static compression with Brotli working for compressing web assets, along the way of getting that worked out, you realise that few people can be bothered with it, hence you are having to compile your own Nginx modules.
I found out the way to do it was to have Nginx run a proxy server that caches the output of another Nginx 'server' that does the Brotli dialled up to 11. So you are making your own CDN.
Nothing is difficult when you know it, but Gzip has scratched the compression itch so well that people just do not have a problem with it and therefore do not seek to change it.
I found out the way to do it was to have Nginx run a proxy server that caches the output of another Nginx 'server' that does the Brotli dialled up to 11. So you are making your own CDN.
Nothing is difficult when you know it, but Gzip has scratched the compression itch so well that people just do not have a problem with it and therefore do not seek to change it.
There is also the cost of a bug. Imagine that AWS corrupted customers data due to a compression bug in your home-grown compression code.
It is almost like encryption in this regard.
It is almost like encryption in this regard.
Luckily by adding a checksum of pre-compression data it's very easy to catch bugs before they do widespread silent damage.
Amazon may be big but even then resources are limited. Individual teams owning the roadmap for services are still small. And there are always competing priorities. Do you prioritize these security and availability improvements or do you pickup some operational improvements to make our engineers oncall easier? Oh there is also this long line of customer features you need to deliver. And re:Invent is also approaching.
When you look at it like this, not very surprising that initiatives like cost savings optimizations may take a back seat for periods of time.
When you look at it like this, not very surprising that initiatives like cost savings optimizations may take a back seat for periods of time.
> I want to know why a company as big as Amazon was still using gzip in 2022....
The short answer to "Why still using gzip in 2022" is almost always answered by "Because it made no business or financial sense to spend head count on it".
Amazon is usually fairly smart about the ways it spends head count, particularly on things that could notably reduce costs. Managers/directors/VPs/SVPs obsess over what the value is from various work, and routinely adjust priorities and reallocate head count. They track not just what is happening within individual services, but also the overall business strategies, what's on the horizon etc.
One simple example that most folks outside the industry won't be seeing is that every major cloud has been working their collective arses off for well over a year on meeting JWCC contract needs. There's a lot of work involved in that contract, because unsurprisingly it's really hard to build and run an entirely air-gapped cloud region, but the payoff on being selected as a vendor is phenomenal. Far more than saving an additional 30% of storage, even at AWS S3 scale. _It's not the only major business and engineering initiative_ that will be taking place across AWS.
I've got a long list of "We should do x, y, z" that applies to my service in the cloud I work for, a number of which will see notable performance improvements. They make zero sense allocating head count to, though, because I've got at least a dozen other more important things going on from a business roadmap perspective to get solved that will make us way more money and/or save way more engineering resources down the line, be it automation stuff, or new features.
What does happen, though, is that list is kept in mind any time new business priorities come about. If there is any way bits on the list can be tied in to something the business wants, it'll get hooked in to it. The business gets what it wants, the service gets what I want, everyone is happy.
(To repeat something others have pointed out, there's also zero indication of timeline, it could have happened any time in the last several years that zstd has been a thing, they may well have switched to it almost as soon as Facebook removed that ridiculous "You can't sue us" clause)
The short answer to "Why still using gzip in 2022" is almost always answered by "Because it made no business or financial sense to spend head count on it".
Amazon is usually fairly smart about the ways it spends head count, particularly on things that could notably reduce costs. Managers/directors/VPs/SVPs obsess over what the value is from various work, and routinely adjust priorities and reallocate head count. They track not just what is happening within individual services, but also the overall business strategies, what's on the horizon etc.
One simple example that most folks outside the industry won't be seeing is that every major cloud has been working their collective arses off for well over a year on meeting JWCC contract needs. There's a lot of work involved in that contract, because unsurprisingly it's really hard to build and run an entirely air-gapped cloud region, but the payoff on being selected as a vendor is phenomenal. Far more than saving an additional 30% of storage, even at AWS S3 scale. _It's not the only major business and engineering initiative_ that will be taking place across AWS.
I've got a long list of "We should do x, y, z" that applies to my service in the cloud I work for, a number of which will see notable performance improvements. They make zero sense allocating head count to, though, because I've got at least a dozen other more important things going on from a business roadmap perspective to get solved that will make us way more money and/or save way more engineering resources down the line, be it automation stuff, or new features.
What does happen, though, is that list is kept in mind any time new business priorities come about. If there is any way bits on the list can be tied in to something the business wants, it'll get hooked in to it. The business gets what it wants, the service gets what I want, everyone is happy.
(To repeat something others have pointed out, there's also zero indication of timeline, it could have happened any time in the last several years that zstd has been a thing, they may well have switched to it almost as soon as Facebook removed that ridiculous "You can't sue us" clause)
How much does it cost to switch? How much effort are you putting into making sure there are no pathological cases while encoding and decoding that have an outsized effect on resource use? How much risk are you willing to take that you don't lose any customer data? If you recompress stored data, how much are you willing to pay in compute? Recompressing at AWS scale is enough IO to burn through many thousands of disks in aggregate—how much does all that wear cost? How much tooling do you need to rewrite?
Let's say you save a few hundred million dollars a year after switching. The cost to switch is almost certainly more than a few hundred million dollars. When you make that kind of investment—especially when you're moving away from a very boring technology—you want to be damn sure you know exactly what you're getting yourself into.
Let's say you save a few hundred million dollars a year after switching. The cost to switch is almost certainly more than a few hundred million dollars. When you make that kind of investment—especially when you're moving away from a very boring technology—you want to be damn sure you know exactly what you're getting yourself into.
The tweet doesnt mention dates or the coverage.
Second I believe youre underestimating the scope of the problem. Amazon has thousands of teams, even more services, each with their own priorities, and innumerable different access patterns, data types, etc. For all practical purposes there is no single “they.”
To get an idea of the scope how long would it take you to remove gz from oh … 50,000 projects? Each with many deployments and up to 15 years of active data.
Second I believe youre underestimating the scope of the problem. Amazon has thousands of teams, even more services, each with their own priorities, and innumerable different access patterns, data types, etc. For all practical purposes there is no single “they.”
To get an idea of the scope how long would it take you to remove gz from oh … 50,000 projects? Each with many deployments and up to 15 years of active data.
Because finishing the next product you can sell can produce more revenue than the savings from switching to zstd.
The beauty of gzip is that it is effective, free, resource efficient, and it works with everything.
A parallel implementation like pigz is still pretty good and widely backwards compatible.
I'm surprised they aren't using an FPGA solution
Knowing such implementation exists, is open source, and since they don't state what they're using... you may just be right
https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany...
https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany...
FPGAs are really expensive in volume compared to other options, so it's not surprising to me.
But on an FPGA you can probably implement super fast and effective compression.
If it is far enough above the curve of speed Vs ratio, then it will pay for itself in saved storage.
Remember there is no need for said FPGA to be in every machine - in a data center with ten's of gigabits of bandwidth to every node, you can send data to another machine for compression and receive back the compressed data to store.
If it is far enough above the curve of speed Vs ratio, then it will pay for itself in saved storage.
Remember there is no need for said FPGA to be in every machine - in a data center with ten's of gigabits of bandwidth to every node, you can send data to another machine for compression and receive back the compressed data to store.
Standard Disclaimer: I work at AWS. Not only are all opinion my own. I’m so far away removed from the service teams, that I might as well be a just like any other customer - I work in the consulting department.
That being said, looking at things from the outside, I tend to think that objects are not stored in a compressed form. It would make many operations much slower.
1. There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object?
https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing...
2. You can query over multiple objects using SQL either via Athena or S3 Select.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-gla...
Edit:
Someone has already explained how byte range requests would work.
That being said, looking at things from the outside, I tend to think that objects are not stored in a compressed form. It would make many operations much slower.
1. There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object?
https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing...
2. You can query over multiple objects using SQL either via Athena or S3 Select.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-gla...
Edit:
Someone has already explained how byte range requests would work.
> There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object?
I wonder if you could store the objects in chunks and then individually compressed. Create an index of the byte ranges for those chunked objects so they can be looked up easily.
For example:
range 0-4000 -> objectA, objectB, objectC
Then just return those 3 objects, decompressed.
I wonder if you could store the objects in chunks and then individually compressed. Create an index of the byte ranges for those chunked objects so they can be looked up easily.
For example:
range 0-4000 -> objectA, objectB, objectC
Then just return those 3 objects, decompressed.
> Someone has already explained how byte range requests would work.
Where?
Where?
Probably a reference to https://news.ycombinator.com/item?id=32531141
Is there some nuance in the phrase “compressed storage” here? Like, is that a subset of overall storage, with some objects not being compressed? I imagine that a great proportion of S3 objects are not amenable to zstd compression, either because they are natively in a compressed format (e.g. JPEG) or because the customer has already compressed them to realise the saving themselves.
I believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.
Hmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?
It is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.
I think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.
It's written through a virtual file system which stores in blocks and not files, the implementation of that could happily run zstd on the way to the multiple physical disks.
This isn't about them shrinking your individual files, its compression on their internal transport.
This isn't about them shrinking your individual files, its compression on their internal transport.
If the customers stored .gz/.zip/... files you could transparently transcompress them to .zstd and back (with an added test of course)
It seems like that would cause problems. Right off the bat, any integrity hashes like md5 or sha256 for the original compressed files would likely be corrupted. Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer. Zip archives in particular can have arbitrary data pretended to them.
I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.
I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.
> Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer.
EPUB is an example of this. They're mostly bog-standard ZIP archives, but in order for the file to be valid, the "first" file in the archive, linearly speaking, must be named "mimetype" and stored with no compression (each file in a ZIP archive can have a different compression level). If you just unzip an EPUB and then just dumbly zip it back up again, your end file will not be a valid EPUB.
EPUB is an example of this. They're mostly bog-standard ZIP archives, but in order for the file to be valid, the "first" file in the archive, linearly speaking, must be named "mimetype" and stored with no compression (each file in a ZIP archive can have a different compression level). If you just unzip an EPUB and then just dumbly zip it back up again, your end file will not be a valid EPUB.
Besides ZIP metadata, there may be flushes in specific points in the deflate stream (which applies to .gz files as well). These flushes reset the compression dictionary and make further compressed data independent from previous data (at the expense of losing some compression efficiency). So: AWS S3 customer may have injected these flushes to their .gz files (for whatever reason, e.g. steganography), and after gzip-to-zstd-to-gzip transcompression this steganographic data will be be lost (and of course sha256 and other similar hashes will be different, as you already said).
Exactly. I've built several logging systems over the years that intentionally flush compression streams (or concatenate gz streams) for robustness reasons.
You would only apply the compression on the internally stored file and then decompress when retrieving it for the customer again. That way all the hashes and original structure of the user are retained.
You would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly.
This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good.
You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly.
I doubt this would even gain you a single percentage.
This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good.
You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly.
I doubt this would even gain you a single percentage.
That wastes a lot of CPU compared to just running it through zstd.
There are tools like preflate [1] or precomp [2] that guarantees a bitwise identical reconstruction, of course modulo bugs.
[1] https://github.com/deus-libri/preflate
[2] https://github.com/schnaader/precomp-cpp/ (which internally makes use of preflate)
[1] https://github.com/deus-libri/preflate
[2] https://github.com/schnaader/precomp-cpp/ (which internally makes use of preflate)
Certainly, if used as a data store for a data-lake-like solution, it's much more convenient to store data in uncompressed form, knowing that the underlying storage solution will take care of that for you. This way, you can just dump Json files in without it feeling like a collosal waste (or having to do extra work of converting them to a compressed format, such as Avro).
Storage level compression usually benefits whomever owns the storage.
I doubt Amazon intends to bill on compressed size.
I doubt Amazon intends to bill on compressed size.
The only way to get the savings yourself is to zstd yourself in the first place, which is what we do.
Not be that guy, but we never send an object to S3 without compressing it on our end first.
We never send anything unencrypted to S3. I have no clue why someone would not want to encrypt their data before putting it in the cloud…
Would you encrypt files that are supposed to be publicly readable and accessible?
I get hashing it and providing hashes, but encrypting public files seems excessive.
I get hashing it and providing hashes, but encrypting public files seems excessive.
For people also using other AWS services (in particular compute ones like EC2) this doesn't do anything meaningful.
We use other aws services. We decrypt the files on those services.
There's a very common data/ETL pattern wherein raw (unencrypted) data is stored into S3 at the very beginning of any pipeline. Adding encryption adds a layer for failure, which can grind your pipeline to a halt.
I've seen a pattern of: drop raw data into an S3 bucket that has a very restrictive policy with a long retention policy. Then, process that data asynchronously (encrypt, transform, filter, etc) and drop it into a different bucket/area that is accessed by other consumers.
Then, if any part of your ETL fails (encryption included), you can fix your bug and reprocess from your raw data without writers seeing any impact.
I've seen a pattern of: drop raw data into an S3 bucket that has a very restrictive policy with a long retention policy. Then, process that data asynchronously (encrypt, transform, filter, etc) and drop it into a different bucket/area that is accessed by other consumers.
Then, if any part of your ETL fails (encryption included), you can fix your bug and reprocess from your raw data without writers seeing any impact.
Because there is already server side encryption. Which I guess means data on disk is always encrypted not sure about in-memory.
We also use server side encryption. But honestly encrypting the files before putting on s3 has given us 0 grief and gives us a feeling even if we somehow are “hacked” the files are useless to the hackers.
Where is your decryption key stored?
At the scale of AWS S3, exabyte scale, I assume that could be close to a billion dollar in cost savings?
Amazon will pay perhaps $50 for a 10TB disk, 10x it to cover for redundancy and the servers and datacenters to put the disks in, and you're looking at "only" 16 mil savings for an exabyte.
Anyone by chance know how many exabytes s3 is?
Did some searches but came up with some guesses like https://maisonbisson.com/post/how-big-is-s3/ but they are dated and still just guesses.
Did some searches but came up with some guesses like https://maisonbisson.com/post/how-big-is-s3/ but they are dated and still just guesses.
Let's say a 100 exabyte, which is only 100 times larger than Backblaze. And Amazon pay $100 per 10TB, it is unrealistically low, but just to round down the number nicely . ( It is not $50 for 10TB, part of the reason why HN dont understand Supply Chains and BOM cost. )
On pure storage that is 100,000,000TB. That is 1 billion dollar already excluding any redundancy required by S3.
On pure storage that is 100,000,000TB. That is 1 billion dollar already excluding any redundancy required by S3.
As a semi-educated guess, I wonder if he might just be referring to the compression of log files via standard Amazon-internal mechanisms, which use S3 as the backing storage, for many services.
Good article on zstd: https://gregoryszorc.com/blog/2017/03/07/better-compression-...
Not too long ago I started compressing statistics data I was collecting from GitHub[1] with ZStandard after noticing that GitHub Actions CI caching used it.
It’s really remarkable for my use case. I compress gigabyte-sized SQLite databases under the GitHub 100 MB file limit since Git LFS is a non-starter. My tables are collections of follower and stargazer counts to measure user and repository popularity and understand where one might be on a distribution compared to a subset of other GitHub data points.
[1]: https://github.com/andrewmcwattersandco/github-statistics
It’s really remarkable for my use case. I compress gigabyte-sized SQLite databases under the GitHub 100 MB file limit since Git LFS is a non-starter. My tables are collections of follower and stargazer counts to measure user and repository popularity and understand where one might be on a distribution compared to a subset of other GitHub data points.
[1]: https://github.com/andrewmcwattersandco/github-statistics
gzip has rsync-friendly mode allowing for much smaller deltas between files during rsync transfer or backups that use rsync-compatible algorithms. No other compressor implements that as far as I know.
zstd 1.3.8 and later also has an rsync-friendly mode (via the --rsyncable command-line option). See https://github.com/facebook/zstd/issues/1155
Nice to know, thanks for pointing this out.
Does it mean that gzip is a legacy compression than should be replaced with zstd where possible?
Yes. Gzip used to be a decent middle ground when you needed neither super fast compression/decompression nor super high compression ratio, but zstd beats it by a good margin over the whole range from gzip -1 to gzip -9, while offering you a wider range of compression ratios to choose from. And at settings that provide compression ratios comparable to gzip it's blazingly fast.
gzip will stay with us because of ubiquity, just as we mostly use image formats from the dot-com era. But if you control both sides, zstd is a big upgrade.
gzip will stay with us because of ubiquity, just as we mostly use image formats from the dot-com era. But if you control both sides, zstd is a big upgrade.
> the whole range from gzip -1 to gzip -9
gzip compression levels are almost useless and typically result in very little actual compression ratio differences, but typically with massive CPU usage differences.
zstd doesn't do everything better than gzip though; I believe gzip still offers somewhat faster compression for equal compression ratios last time I looked (but decompression is much faster). I mostly replaced gzip with zstd myself, but there are still scenarios where gzip might be preferable.
gzip compression levels are almost useless and typically result in very little actual compression ratio differences, but typically with massive CPU usage differences.
zstd doesn't do everything better than gzip though; I believe gzip still offers somewhat faster compression for equal compression ratios last time I looked (but decompression is much faster). I mostly replaced gzip with zstd myself, but there are still scenarios where gzip might be preferable.
In all the testing I have been doing across a variety of use cases, from BigData, to databases, to games, I have not come across a scenario where Zstd is doing worse than Zstd.
Compression time CPU is lower for equivalent ratios. Maximum compression is better. Decompression CPU is much lower (i.e. faster), and that is independent of the level used at compression time.
Compression time CPU is lower for equivalent ratios. Maximum compression is better. Decompression CPU is much lower (i.e. faster), and that is independent of the level used at compression time.
I did a bunch of testing about five years ago and in some scenarios gzip was a bit faster, but I don't recall which exactly and it was five years ago so things may have changed. I thought I had saved the results somewhere, but I can't find them right now.
Overall, zstd still came out as the clear winner (also compared some other compression tools), so I went with that.
Overall, zstd still came out as the clear winner (also compared some other compression tools), so I went with that.
Yes, if you control both sides, always use zstd instead of deflate/gzip/zip.
[deleted]
I think he means a 30% reduction in used storage and not that 30% of objects were no longer being stored.
It would be a reduction in data, but not compressed data.
It would be a reduction in data, but not compressed data.
I'd be really curious to hear the security discussion around S3 compression, or if they added any additional mitigations. Compressing data and then encrypting it will be vulnerable to a padding oracle attack[1].
[1] https://en.m.wikipedia.org/wiki/CRIME
[1] https://en.m.wikipedia.org/wiki/CRIME
But they still only support gzip or bzip2 for S3 Select’ing json files :(
If relevant for you, we have an unadvertised feature in MinIO, that adds Zstandard, LZ4, Snappy support:
https://github.com/minio/minio/blob/master/internal/s3select...
https://github.com/minio/minio/blob/master/internal/s3select...
Random shallow rant, I really wish they came up with a different abbreviation than zstd. Not only it sounds strange but it’s also too long for abbreviation, imo.
It could have been “zs” or “zd”.
Quick question, for personal file storage, such as family photos, would zstd be preferable now to, say, tar.gz?
Photos generally won't get much compression anyways since the image data is already compressed. However in general yes, zstd is almost always preferable to gzip these days.
For archiving RAW files, yes.
For RAW --> lossy format, storing them in JPEG-XL would be preferable, and I doubt most compression algos would do better than JPEG-XL.
For RAW --> lossy format, storing them in JPEG-XL would be preferable, and I doubt most compression algos would do better than JPEG-XL.
zstd not good for everything. Yes, it's fast in terms of compression. Easy & fast you can make compressed archive. But if you need to use the zstd on daily basis - i have bad news. ZSTD not suitable for general purpose archiver.
Depends on the data - you can have totally different output compressed archive if compare to winrar for example.
Example: I doing backups time to time, and archive important files into compressed archive containers (rar / 7zstd).
I've noticed, that my dev folder with tons of repositories, images, and different work related fines - vary damn too much.
/dev/ size = ~11GB
rar output (normal compression) = ~2.1 GB zstd archive output (normal compression) = ~4.7GB
Why? linked files, same files not treated as a 1 file + links to these files. Instead these files compressed each 1 by 1 instead of copy 1 identical, and compress the file. And many things like that.
Suggestion for 7z-zstd -> add ability to save links to files, not treat them as separate files, and adding an option like in winrar to search for identical files first and re-link all of them and remove duplicates, instead of compression each.
Depends on the data - you can have totally different output compressed archive if compare to winrar for example.
Example: I doing backups time to time, and archive important files into compressed archive containers (rar / 7zstd).
I've noticed, that my dev folder with tons of repositories, images, and different work related fines - vary damn too much.
/dev/ size = ~11GB
rar output (normal compression) = ~2.1 GB zstd archive output (normal compression) = ~4.7GB
Why? linked files, same files not treated as a 1 file + links to these files. Instead these files compressed each 1 by 1 instead of copy 1 identical, and compress the file. And many things like that.
Suggestion for 7z-zstd -> add ability to save links to files, not treat them as separate files, and adding an option like in winrar to search for identical files first and re-link all of them and remove duplicates, instead of compression each.
Your criticisms seem to be about 7z, not zstd. Ztd doesn't have an archive concept, it's just compresses a sequence of bytes.
i'm totally correct in what am i writting. zstd not wide range suitable compression. And all current implementation that used non for 1 file compression - awful.
No, your critiques are of how 7zip implements an archival format built on top of zstd. The zstd algorithm has no concept of files, only bytes.
Archival software then has to build a file format on top of the compression algorithm, and there are multiple ways to slice the problem. For example, a tar.gz will first tar everything into a big archive file, then feed it into gzip for compression. zip, on the other hand, feeds each file individually into the chosen algorithm (DEFLATE for most implementations).
Your critique is that the 7zip archive format is not suitable for use with zstd in the case of many small yet identical files. zstd is doing its job, just the archival format is not playing along.
Archival software then has to build a file format on top of the compression algorithm, and there are multiple ways to slice the problem. For example, a tar.gz will first tar everything into a big archive file, then feed it into gzip for compression. zip, on the other hand, feeds each file individually into the chosen algorithm (DEFLATE for most implementations).
Your critique is that the 7zip archive format is not suitable for use with zstd in the case of many small yet identical files. zstd is doing its job, just the archival format is not playing along.
Zstandard doesn't accept multiple files at all, so it's an archiving format's job to convert files into byte sequences and compress them accordingly. It looks like that 7z wasn't able to deduplicate entirely or partially identical files, in the other words zstd could never know that it is compressing almost identical files over and over.
No support in CompressionStream, that's too bad.
But what about the % increase in power consumption?
Zstd is very efficient in decompression https://engineering.fb.com/2016/08/31/core-data/smaller-and-...
Everything counts in large amounts.
Which of course will not be passed to the consumer
Well yeah. S3 is ~$23/TB/month vs $10/TB HDD NRC + $0.20/TB/month electricity for a data hoarder NAS. I can’t imagine S3 users are very price sensitive. Most of them store <1PB.
Of course there are cheaper cloud services like B2 at $5/TB/mo, but even that can’t compete on price with physical infra when you have more than a handful of drives.
Of course there are cheaper cloud services like B2 at $5/TB/mo, but even that can’t compete on price with physical infra when you have more than a handful of drives.
Not that it changes the math much, but:
$15/TB is hard enough for a home buyer to reach for hard drives, excluding server cost. Where did you get $10?
$15/TB is hard enough for a home buyer to reach for hard drives, excluding server cost. Where did you get $10?
Not them, but searching for "8TB HDD" on Amazon (huh, ironic) shows several options for $90, which is almost $10/TB. And this was 60 seconds of searching, no real effort put in.
$15/TB is extremely expensive if you’re building a raidz2 array.
$9.70/TB 10TB: https://www.ebay.com/itm/275400447467
$8.31/TB 4x8TB: https://www.ebay.com/itm/125132232253
You can get even cheaper if you buy in bulk.
Server cost is $1/TB with reasonable density
$9.70/TB 10TB: https://www.ebay.com/itm/275400447467
$8.31/TB 4x8TB: https://www.ebay.com/itm/125132232253
You can get even cheaper if you buy in bulk.
Server cost is $1/TB with reasonable density
Oh. Used drives.
they should at least etch our names into the yacht that the savings will buy
As a cloud employee I am willing to make the public commitment that should I ever be in the position of buying a yacht, I will etch "booi" into the hull for you.
[deleted]
That's what competition is for.
Google/Azure should compress and pass the savings back to the customer..
Well if they figure they will get more customers to offset lower profits they’ll reduce the price. On the other hand if they think they can double the price but only lose 10% of customers they’ll do that
It’s econ101, the cost of providing the service doesn’t really factor into the price charged, what does factor in is the cost of alternatives (storage from azure, backblaze, on prem, not doing it), and the risks (increase s3 costs and people might realise they are being ripped off on ec2)
It’s econ101, the cost of providing the service doesn’t really factor into the price charged, what does factor in is the cost of alternatives (storage from azure, backblaze, on prem, not doing it), and the risks (increase s3 costs and people might realise they are being ripped off on ec2)
Yes. The cost of providing the service only factors indirectly into the price charged: via competition.
I'm not holding my breath
Zstd, it’s okay. 600 .rvz files = 450 Gb -> 163 GB. Wanted it more like 10 GB.
I don't think AWS routinely compress customer data that I've noticed. I guess he must mean for their internal products that use S3 perhaps?