Looks like a big deal. Is this basically paving the way to free cross-cloud connectivity?
I wonder why they would only do this for private networks and not for public traffic. For example, AWS is still not part of the Bandwidth Alliance [1], so if you put a 3rd party CDN in front of AWS you can still end up paying a lot more in egress fees compared to using their own offerings.
Maybe private interconnects are just much easier to cap/control/attribute, but it feels like an interesting split.
That's a clean architecture and the dataset abstraction isn't really needed when every file has a DB row and clear lifecycle.
The post is more about the pipeline / ML / log / export world where ownership isn't enforced by application code.
The DMS case sits somewhere in between - there's a per-table grouping that could be useful, but the files are usually transient enough that it doesn't matter much.
Different problem from yours.
Ha. The fact that teams reach for iceberg to organize things that aren't really tables is itself a symptom of needing better management tools for other types of data.
Curious what your use cases look like. If you're storing data where you always know what's there, who created it, and whether it's still in use without needing to query for it, that's actually a great place to be. The post is about the much messier middle ground most teams I've talked to are in.
They solve it, partially, for tabular data. Delta, Iceberg, DuckLake are all table formats. And yeah, they do more than dataset abstraction (transactions, time travel, schema evolution).
But that's just one slice of storage. Most teams also have logs, media, ML artifacts, raw dumps, etc., none of which fit into a table format. And even with tables, you often can't easily look at a Delta table and know what the underlying storage is costing you, whether it's still accessed, etc.
Another system might solve it for your media files, another for your log streams, and so on. That's the thing, you have a set of management nice-to-haves that are quite generic and aren't universally supported today, so you end up reinventing them separately across each domain. And even if you did, you still wouldn't have a central aggregated view across all your storage.
You can have lots of buckets, but each one typically still contains many datasets.
Think of a team doing ML, for example. They work with data all day across many different tools, each reading some inputs from S3 and writing outputs to S3. They won't create a bucket for every output, that's not practical. So they write to a single bucket with outputs organized under prefixes.
Buckets are more of an administrative boundary (IAM, cost, replication) than a data organization unit. So even with more buckets, the dataset abstraction is still missing - there's no good native way to track what a prefix represents, who created it, whether it's still accessed, how much it costs, etc.
Good point, prefixes are performance boundaries too, per-prefix rate scaling means you can spread load across prefixes to get aggregate throughput well above 3.5k RPS [1].
But that's a different thing than what the post is about. Even teams that use prefixes for performance don't have an S3-native way to ask what a prefix represents, who owns it, whether it's still accessed, and so on. The semantic layer is missing whether you're hashing for throughput or just laying data out the obvious way.
Agree it doesn't have to be part of S3 itself. My point is that there is a missing semantic layer.
In practice, many teams use S3 directly without any layer on top. So without better organizational capabilities, they can't keep track of what they have stored where, who created it, whether it is still used, etc.
And when teams do use a catalog, it's usually detached from the storage layer itself, so you can't easily view a dataset in the catalog and know how much it costs, who accessed it, and so on.
Have you seen better places that figured out a better way to handle this? Without a ton of custom tooling?
I wonder why they would only do this for private networks and not for public traffic. For example, AWS is still not part of the Bandwidth Alliance [1], so if you put a 3rd party CDN in front of AWS you can still end up paying a lot more in egress fees compared to using their own offerings.
Maybe private interconnects are just much easier to cap/control/attribute, but it feels like an interesting split.
[1] https://www.cloudflare.com/bandwidth-alliance/