Apache Hudi vs. Delta Lake vs. Apache Iceberg Lakehouse Feature Comparison(onehouse.ai)
onehouse.ai
Apache Hudi vs. Delta Lake vs. Apache Iceberg Lakehouse Feature Comparison
https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-apache-iceberg-lakehouse-feature-comparison
54 comments
I think the blog post should point out very early that Onehouse is a Hudi company. There are some other recent benchmarks published in CIDR by Databricks that might paint a different picture: https://petereliaskraft.net/res/cidr_lakehouse.pdf
Thanks for the link. I'd be interested to see a perf comparison using a popular processing engine other than spark given the obvious potential for delta lake to be better tuned for spark workloads by default.
me too. Trino for one would be a good start. Adding support for those data lakes is really hard though if you want good performance.
In Databricks published benchmark of course Delta is the fastest. I have also seen some Iceberg using company publishing benchmarks showing how Iceberg is the fastest.
Vendor published benchmarks are worthless.
Vendor published benchmarks are worthless.
I think vendor published benchmarks are fine if the dataset is open / accessible, the benchmark code is published, all software versions are disclosed, and the exact hardware is specified. I definitely wouldn't consider an audited TPC benchmark that's based on industry standard datasets / queries worthless in the data space. Disclosure: I work for Databricks.
fwiw - the lead authors on that linked paper are all grad students not employed at Databricks. That being said, they're advised by Databricks people
It looks like the benchmarks used the latest versions of Delta and Iceberg, but chose a version of Hudi that is over 6 months old. Hudi v0.12.2 is more advanced than v0.12.0 which the benchmark did not consider. As the Databricks CIDR paper states, and as mentioned in the Onehouse article, Hudi by default is optimized for UPSERTs vs INSERTs and is a 1-line config change that is appropriate for a true apples-apples comparison. See both: https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-trans... and https://github.com/brooklyn-data/delta/pull/2
Hah, I could tell because in the "feature matrix" the Hudi column was mostly green compared to the others. Immediately made me suspicious so I looked it up and sure enough, not exactly an unbiased source.
Feature matrices are extremely easy to game depending on your choice of rows.
Feature matrices are extremely easy to game depending on your choice of rows.
I recently evaluated these frameworks and went through all these links they have for each of those rows, on the first publish few months ago. FWIW I did not find any inaccuracies or wrong pointers.
Thank you, for that I’m sure entirely unbiased assessment on your newly-created account.
Feature matrices are fundamentally flawed for the reason that the GP gave.
Feature matrices are fundamentally flawed for the reason that the GP gave.
it's funny how, on one hand you argue for objectivity but fundamentally distrust/write off a chance that someone could have created a hackernews account today and comment here - without a shred of evidence. May be now I am getting trained on the HN ways.
Some high level context for those less familiar with the Lakehouse storage system space. For various reasons, several companies moved from data warehouses to data lakes starting around 7-10 years ago.
Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data.
Data lakes were limited in many regards. They were easily corrupted (no schema enforcement), required slow file listings when reading data, and didn't support ACID transactions.
I'm on the Delta Lake team and will speak to some of the benefits of Delta Lake compared to data lakes:
* Delta Lake supports ACID transactions, so Delta tables are harder to corrupt. The transaction log makes it easy to time travel, version datasets, and rollback to earlier versions of your data.
* Delta Lake allows for schema enforcement & evolution
* Delta Lake makes it easy to compact small files (big data systems don't like an excessive number of small files)
* Delta Lake lets readers get files and skip files via the transaction log (much faster than a file listing). Z ORDERING the data makes reads even faster.
The Delta Lake protocol is implemented in a Scala library and exposed via PySpark, Scala Spark, and Java Spark bindings. This is the library most people think of when conceptualizing Delta Lake.
There is also a Delta Lake Java Standalone library that's used to build other readers like the Trino & Hive readers.
The Delta Rust project is another implementation of the Delta Lake protocol that is implemented in Rust. This library is accessible via Rust or Python bindings. Polars just added a Delta Lake reader with delta-rs and this library can also be used to easily read Delta Lakes into other DataFrames like pandas or Dask.
Lots of DataFrame users are struggling with data lakes / single data files. They don't have any data skipping capabilities (unless Parquet file footers are read), their datasets are easily corruptible, and they don't have any schema enforcement / schema evolution / data versioning / etc. I expect the data community to accelerate the shift to Lakehouse storage systems as they learn about all of these advantages.
Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data.
Data lakes were limited in many regards. They were easily corrupted (no schema enforcement), required slow file listings when reading data, and didn't support ACID transactions.
I'm on the Delta Lake team and will speak to some of the benefits of Delta Lake compared to data lakes:
* Delta Lake supports ACID transactions, so Delta tables are harder to corrupt. The transaction log makes it easy to time travel, version datasets, and rollback to earlier versions of your data.
* Delta Lake allows for schema enforcement & evolution
* Delta Lake makes it easy to compact small files (big data systems don't like an excessive number of small files)
* Delta Lake lets readers get files and skip files via the transaction log (much faster than a file listing). Z ORDERING the data makes reads even faster.
The Delta Lake protocol is implemented in a Scala library and exposed via PySpark, Scala Spark, and Java Spark bindings. This is the library most people think of when conceptualizing Delta Lake.
There is also a Delta Lake Java Standalone library that's used to build other readers like the Trino & Hive readers.
The Delta Rust project is another implementation of the Delta Lake protocol that is implemented in Rust. This library is accessible via Rust or Python bindings. Polars just added a Delta Lake reader with delta-rs and this library can also be used to easily read Delta Lakes into other DataFrames like pandas or Dask.
Lots of DataFrame users are struggling with data lakes / single data files. They don't have any data skipping capabilities (unless Parquet file footers are read), their datasets are easily corruptible, and they don't have any schema enforcement / schema evolution / data versioning / etc. I expect the data community to accelerate the shift to Lakehouse storage systems as they learn about all of these advantages.
Not that I know what anything means in "big data lake OLAP database" anymore, but I always thought a data lake implied a lot of hybrid sources/formats/structures for the data, but the advocacy here implies that the data is all ingested and reformatted, which to me is a data warehouse.
But then again, data lake may simply be what a data warehouse is now called in marketspeak.
Also, I stopped paying attention when the treadmill of new frameworks became unbearable to track, is spark now settled as the standard of distributed "processing", as in mapreduce / distributed query / distributed batch / etc?
I get that performance can improve by unifying to a file format like parquet, but again that seems like a data warehouse. A data lake should be something over heterogenous sources with "drivers" or "adaptors" IMO, in particular because the restoration of the data inputs stays in the knowledge domain of the source production database maintainers.
But then again, data lake may simply be what a data warehouse is now called in marketspeak.
Also, I stopped paying attention when the treadmill of new frameworks became unbearable to track, is spark now settled as the standard of distributed "processing", as in mapreduce / distributed query / distributed batch / etc?
I get that performance can improve by unifying to a file format like parquet, but again that seems like a data warehouse. A data lake should be something over heterogenous sources with "drivers" or "adaptors" IMO, in particular because the restoration of the data inputs stays in the knowledge domain of the source production database maintainers.
You're understandably confused by the industry terminology that's ambiguous and morphing over time.
Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data).
A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created.
And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.
Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data).
A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created.
And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.
You're making a lot of assertions I am not sure I agree with:
> Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data.
- I am not sure it's any cheaper than BQ or Snowflake storage.
- Modern CDW separates compute from storage.
- I am not sure what you mean by "you need to share compute with others". Why?
- You can attach an arbitrary number of "clusters" in BQ and Snowflake as well.
Additionally, modern CDW provides a very high level of abstraction and a very high level of manageability. Their time travel and compaction actually work, and their storage systems are continuously optimized for optimal performance.
> Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data.
- I am not sure it's any cheaper than BQ or Snowflake storage.
- Modern CDW separates compute from storage.
- I am not sure what you mean by "you need to share compute with others". Why?
- You can attach an arbitrary number of "clusters" in BQ and Snowflake as well.
Additionally, modern CDW provides a very high level of abstraction and a very high level of manageability. Their time travel and compaction actually work, and their storage systems are continuously optimized for optimal performance.
Isn't it ACID transactions on ... one table? I thought joins were right out for that on DLT at this point.
In our use case (rebuilding tables from Postgres CDC logs), Delta Lake was significantly faster than Hudi.
Are you open to sharing more? We'd love to highlight this success and contribute any learnings back to the community!
nick ( a t ) nickkarpov.com
nick ( a t ) nickkarpov.com
The data lakehouse space is the most exciting evolution of the data warehousing industry imo! If anyone wants a comparison guide from a neutral party, we did one last year: https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b...
I cant paste images here but imo this table comparing the 3 formats is the big takeaway https://assets-global.website-files.com/6064b31ff49a2d31e049... (explained inline, we do cite onehouse heavily but we are independent of them)
I cant paste images here but imo this table comparing the 3 formats is the big takeaway https://assets-global.website-files.com/6064b31ff49a2d31e049... (explained inline, we do cite onehouse heavily but we are independent of them)
joeharris76(1)
These formats look like an attempt to get a halfway solution: you want to get something like a real MPP analytic DBMS (e.g., ClickHouse) but have to use a data lake for some reason.
It resembles previous trendy technologies that are mostly forgotten now, such as:
- Lambda architecture (based on a wrong assumption that you cannot have a real-time and historical layers in the same system);
- Multidimensional OLAP (based on a wrong assumption that you cannot do analytic queries directly on non-aggregated data);
- Big data (based on a wrong assumption that map-reduce is better than relational DBMS).
I'm exaggerating a little.
Disclaimer: I work on ClickHouse, and I'm a follower of every technology in the data processing area.
It resembles previous trendy technologies that are mostly forgotten now, such as:
- Lambda architecture (based on a wrong assumption that you cannot have a real-time and historical layers in the same system);
- Multidimensional OLAP (based on a wrong assumption that you cannot do analytic queries directly on non-aggregated data);
- Big data (based on a wrong assumption that map-reduce is better than relational DBMS).
I'm exaggerating a little.
Disclaimer: I work on ClickHouse, and I'm a follower of every technology in the data processing area.
The formats are kind of a halfway solution, because trying to build something with MPP semantics on objects stores is difficult.
The difference between MPP and something like Databricks or Trino working with object store is that while MPP can likely get much better performance and especially latency from the same hardware, operating it is much harder.
You don't "backup" Databricks - the data is stored in object storage and that is it. You don't have to plan storage sizing quarters upfront, and you never get in trouble because there is unexpected data spike. Compute resizes are trivial, there is no rebalancing. Upgrades are easy, because you're just upgrading the compute and you can't break data that way. You can give each user group (like batch one and interactive one, or each team) their dedicated compute over common data and it works. That compute can spin up and down and autoscale to save some money. You don't have to think about how to replicate my table across a cluster or anything. And so on, and so forth.
Running a big data and analytics platform - place where tens of teams, tens of applications and hundreds or thousands of analysts come for data and where they build their solutions - is already enough of a challenge without all this operations work, and that is why Snowflake and Databricks are worth that crazy money.
If someone could solve the challenge of having MPP that is as easy to manage as Snowflake or a Lakehouse, that would be quite the differentiator. And maybe you people already did and I just didn't notice, I don't know :)
The difference between MPP and something like Databricks or Trino working with object store is that while MPP can likely get much better performance and especially latency from the same hardware, operating it is much harder.
You don't "backup" Databricks - the data is stored in object storage and that is it. You don't have to plan storage sizing quarters upfront, and you never get in trouble because there is unexpected data spike. Compute resizes are trivial, there is no rebalancing. Upgrades are easy, because you're just upgrading the compute and you can't break data that way. You can give each user group (like batch one and interactive one, or each team) their dedicated compute over common data and it works. That compute can spin up and down and autoscale to save some money. You don't have to think about how to replicate my table across a cluster or anything. And so on, and so forth.
Running a big data and analytics platform - place where tens of teams, tens of applications and hundreds or thousands of analysts come for data and where they build their solutions - is already enough of a challenge without all this operations work, and that is why Snowflake and Databricks are worth that crazy money.
If someone could solve the challenge of having MPP that is as easy to manage as Snowflake or a Lakehouse, that would be quite the differentiator. And maybe you people already did and I just didn't notice, I don't know :)
It would be good if you labeled your posts so as to reveal your bias.
I understand why folks want options. At the end of the day, folks want an easy to use, ALWAYS CORRECT stable database, with minimal well-documented predictable knobs, correct distributed execution plan, no OOMs, separation of storage and compute, and standard SQL, and Clickhouse struggles with all of the above.
(co-founder of MotherDuck)
I understand why folks want options. At the end of the day, folks want an easy to use, ALWAYS CORRECT stable database, with minimal well-documented predictable knobs, correct distributed execution plan, no OOMs, separation of storage and compute, and standard SQL, and Clickhouse struggles with all of the above.
(co-founder of MotherDuck)
We are also interested in the comparison of Hudi, Delta Lake, and Iceberg.
We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integra...
It is a read-only implementation: ClickHouse can read and process the external data in the Hudi or Delta Lake format.
Apache Iceberg is pending. There is no good C++ library for it. But at least the overall structure is simple, as it is not hard to implement it.
The overall principle - whatever data format it is, ClickHouse should support it in a fast, stable, and ALWAYS CORRECT manner.
If you have more experience to share, please do it.
We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integra...
It is a read-only implementation: ClickHouse can read and process the external data in the Hudi or Delta Lake format.
Apache Iceberg is pending. There is no good C++ library for it. But at least the overall structure is simple, as it is not hard to implement it.
The overall principle - whatever data format it is, ClickHouse should support it in a fast, stable, and ALWAYS CORRECT manner.
If you have more experience to share, please do it.
> We have recently added support for Hudi and Delta Lake; you can check here: https://clickhouse.com/docs/en/engines/table-engines/integra...
Just tried it out. It seems to only support partitioned Delta tables, non-partitioned return CANNOT_EXTRACT_TABLE_STRUCTURE. Is that on purpose or is it a bug?
Just tried it out. It seems to only support partitioned Delta tables, non-partitioned return CANNOT_EXTRACT_TABLE_STRUCTURE. Is that on purpose or is it a bug?
Could you please elaborate on your comments and possible misconceptions about ClickHouse? Proven stability, massive scale, predictability, native SQL, and industry-best performance are all well-recognized characteristics of clickhouse, so your comments here seem a bit biased.
I am interested to learn more about your point of view, as well as tangentially the strategic vision of MotherDuck as a company.
(VP Support at ClickHouse)
I am interested to learn more about your point of view, as well as tangentially the strategic vision of MotherDuck as a company.
(VP Support at ClickHouse)
Speaking from nearly a decade working on BigQuery, and a year working at Firebolt.
- Stability. It OOMS, your CTO mentioned that last week.
- It is not correct. I believe your team is aware of cases in which your very own benchmarks revealed Clickhouse to be incorrect.
- Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle.
- SQL. It is very non-standard.
- Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most.
Don't get me wrong, I love open source, and I love what Clickhouse has done. I am not a fan of overselling. There are problems with Clickhouse. Trying to sell it as a superset of the modern CDW is not doing users any favors.
- Stability. It OOMS, your CTO mentioned that last week.
- It is not correct. I believe your team is aware of cases in which your very own benchmarks revealed Clickhouse to be incorrect.
- Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle.
- SQL. It is very non-standard.
- Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most.
Don't get me wrong, I love open source, and I love what Clickhouse has done. I am not a fan of overselling. There are problems with Clickhouse. Trying to sell it as a superset of the modern CDW is not doing users any favors.
As an engineer who admires the work done by DuckDB, I'm disappointed that the co-founder of its evolution is spreading FUD about competitors before its even in the competitive conversation.
> Stability. It OOMS, your CTO mentioned that last week.
I ran ClickHouse clusters for years with zero stability issues (even as a beginner at the time) at an extremely large volume video game studio with real-time needs. Using online materialized views, I was able to construct rollups of vital KPIs at millisecond level while maintaining multi-thousand QPS. Stability was never a concern of ours, and quite frankly, we were kind of blown away.
> Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle.
First, I hate the word "broken" with zero explanation what you mean by this. Based on your language, I'm assuming you're just suggesting the distributed plans aren't as efficient as possible, a limitation that the engineers are not shy to admit.
> SQL. It is very non-standard.
I would argue the language is more a superset than "non-standard". Most everything for us just worked, and often I found areas of SQL that I could reduce significantly due to the "non-standard" extras they've added. For example: Did you know they have built-in aggregate functions for computing retention?!
> Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most.
Yes, there are a lot of knobs. ClickHouse works wonderfully out of the box with the default knobs, but you're free to tinker because that's how flexible the technology is.
You worked at Google for over a decade? You should know. Google's tech is notorious for having a TON of knobs for their internal technology (e.g. BigTable). Just because the knobs are there doesn't mean they must be tuned, it just means the engineers thought ahead. Also, the vast majority of configuration changes I've made never required a restart...I'm not even sure why you pointed this out.
(Disclaimer: I have been using ClickHouse successfully for several years)
> Stability. It OOMS, your CTO mentioned that last week.
I ran ClickHouse clusters for years with zero stability issues (even as a beginner at the time) at an extremely large volume video game studio with real-time needs. Using online materialized views, I was able to construct rollups of vital KPIs at millisecond level while maintaining multi-thousand QPS. Stability was never a concern of ours, and quite frankly, we were kind of blown away.
> Scale. The distributed plan is broken and I'm not sure Clickhouse even has shuffle.
First, I hate the word "broken" with zero explanation what you mean by this. Based on your language, I'm assuming you're just suggesting the distributed plans aren't as efficient as possible, a limitation that the engineers are not shy to admit.
> SQL. It is very non-standard.
I would argue the language is more a superset than "non-standard". Most everything for us just worked, and often I found areas of SQL that I could reduce significantly due to the "non-standard" extras they've added. For example: Did you know they have built-in aggregate functions for computing retention?!
> Knobs. Lots of knobs that are poorly documented. It's unclear which are mandatory. You have to restart for most.
Yes, there are a lot of knobs. ClickHouse works wonderfully out of the box with the default knobs, but you're free to tinker because that's how flexible the technology is.
You worked at Google for over a decade? You should know. Google's tech is notorious for having a TON of knobs for their internal technology (e.g. BigTable). Just because the knobs are there doesn't mean they must be tuned, it just means the engineers thought ahead. Also, the vast majority of configuration changes I've made never required a restart...I'm not even sure why you pointed this out.
(Disclaimer: I have been using ClickHouse successfully for several years)
Funny to read this since based on my testing so far DuckDB is not quite rockstable yet, requires preloading the httpfs module and setting 5 parameters each tun just to query a file off S3 (often very slowly, mostly because of the folder crawling/list logic?) with non standard SQL for extensions and segfaults frequently during testing if the remote server has any unexpected response (also not working too well with S3 clones). The WASM version differs in s3 settings, too but this is not well documented. If you really love opensource and with such a long career you should know there is no point in badmouthing other projects in order to achieve success, particularly from leadership positions. Just make DuckDB great and let happy users speak on your behalf. Nobody wants to be part of a toxic community this early in.
I do NOT work for ClickHouse, but I've been running super stable distributed CH clusters for years.
I do NOT work for ClickHouse, but I've been running super stable distributed CH clusters for years.
Funny noticing this same post pattern by Mother Duck in other threads. It's not a good look.
It is quite surreal that in addition to the usual Databricks astroturfing and various fans arguing, we now also have founders and VPs of companies arguing about who is better on HN.
[deleted]
Naming this concept a 'data lake' bugs me. Turns out when you get old, you read things sometimes and you're thinking "ah, so that's where my line was. Interesting."
You need a data lake before you can build your data lakehouse, of course.
And you need a data lakehouse so you can enjoy your data jetskis.
Some folks use data lakehouses for data fishing or data powersports. Personally I just like to sit out and enjoy the data sunsets on the warm nights.
But one thing to be aware of if your data gets big is that (to paraphrase the film) “We’re gonna need a bigger data lakehouse”.
But one thing to be aware of if your data gets big is that (to paraphrase the film) “We’re gonna need a bigger data lakehouse”.
It’s marketing BS designed to look good as a whiteboard diagram. “See all your disparate data sources here? That’s your problem. Your competitors have all their data in a data warehouse. You like warehouses? No? Neither do I. Stuffy, stale old places. You need all this data to flow into a central living breathing data lake. You like lakes? You do? Well now one thing to be aware of is to really appreciate the data lake, there’s nothing like a data lakehouse…”
Data Lakehouse is where you store your data on object stores and spin up a bunch of instances (cpu/memory) as needed to crunch the data on the timeline you desire. It’s incredible to me how “solutions” are continually invented that give cloud providers plenty of stages to charge for not only storage but movement and processing of your data too.
No, that's data lake. Data lakehouse is data lake where your objects are wrapped in a "table format" like Iceberg that allows you to query (update, itd) them like they were stored in a traditional data warehouse.
The data is still stored on object storage. It’s just a more performant format.
The root of the problem is using object storage improperly.
The root of the problem is using object storage improperly.
Data Lakehouse to me is:
> A Data Lakehouse open data management architecture that combines the flexibility, cost-efficiency, and scale of Data Lake with the data management and ACID transactions of Data Warehouse with Data Lake Table Formats (Delta Lake, Apache Iceberg & Apache Hudi) that enable Business Intelligence (BI) and Machine Learning (ML) on all data.
The initial concept was created by Databricks in the CIDR Paper (https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) in 2021.
The initial concept was created by Databricks in the CIDR Paper (https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) in 2021.
Format is the same - parquet. Only the way metadata is stored is different.
>The root of the problem is using object storage improperly.
I don't see anything improper for any Parquet-with-metadata format.
>The root of the problem is using object storage improperly.
I don't see anything improper for any Parquet-with-metadata format.
Why would you be charged for movement in this case, I thought intra datacenter traffic was free? Or you mean you get charged to update/query the object store?
As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?
Disaggregated storage. You still have separation of compute and storage with the bandwidth and latency benefits of directly attached storage.
Object stores are a terribly inefficient way to access and store changes of data.
Object stores are a terribly inefficient way to access and store changes of data.
It's definitely inefficient w/r/t performance (reading from Object Storage will always be several times slower than reading from disk/SSD) but the point is usually to minimize cost.
Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store
- someone who thinks it’s often a bad idea
- someone who thinks it’s often a bad idea
it's not that it is inefficient, it is that data processing approach has to be different with lakehouses.
you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability.
for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.
you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability.
for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.
I guess a setup such as hdfs where storage and compute are colocated and not disaggregated. But that also offers similar transactional semantics to lakehouses.