A database kernel engineer's take on Apache Iceberg(mooncake.dev)
mooncake.dev
A database kernel engineer's take on Apache Iceberg
https://www.mooncake.dev/blog/iceberg-is-terrible
4 comments
> Even if people somehow get this to work like a real database, they will quickly become frustrated with the same features that the data lake avoids: access controls, locks, consistency, centralization, a schema, you name it.
Are you saying that Apache Iceberg does not have these features? Other than access controls, most of them are there. What Iceberg most certainly does not have is efficient update--you have to load in large batches to avoid metadata file explosion. That's actually fine for cold storage, without which analytic databases become economical at scale. And Iceberg does have features like time travel, which databases like ClickHouse lack. It's also shareable, it's not just an appendage of one particular data warehouse.
One real problem with Iceberg is that a lot of people (vendors mostly) are jumping on it as a way to market their products.
Are you saying that Apache Iceberg does not have these features? Other than access controls, most of them are there. What Iceberg most certainly does not have is efficient update--you have to load in large batches to avoid metadata file explosion. That's actually fine for cold storage, without which analytic databases become economical at scale. And Iceberg does have features like time travel, which databases like ClickHouse lack. It's also shareable, it's not just an appendage of one particular data warehouse.
One real problem with Iceberg is that a lot of people (vendors mostly) are jumping on it as a way to market their products.
author here. I would actually second you.
My core belief is "it is possible to build true database-like functionalities on top of iceberg", but it is definitely not 'easier' than building them directly in a db (in fact, doing them while keep iceberg-compatible is tricky. yep, that's the cost of being open and general purpose)
I've been reading up on lakehouse stuff, and everything seems to agree with the idea that data warehouse functionality deteriorates as they try to accommodate raw data access.
The databricks people suggest that a better "cache format" is needed, but I don't see how that's different from ETLing the data into a regular warehouse.
The databricks people suggest that a better "cache format" is needed, but I don't see how that's different from ETLing the data into a regular warehouse.
But I constantly underestimate the tendency to take a minor capability and exaggerate it into a replacement for 50 years of heavily studied technology.
Even if people somehow get this to work like a real database, they will quickly become frustrated with the same features that the data lake avoids: access controls, locks, consistency, centralization, a schema, you name it.