The commercial product, Foundry, is very well documented and an extensive Data Platform that allows to build data pipelines (similar to Databricks) and build low code / no code applications on top. If you master it, its incredibly powerful but complex
I will never understand how people honestly think that there is a such a thing as a central DB. Do you really think that Gov Agencies from all over the world deploy Gotham just connected to the internet without controlling inflow / outflow of data? I would bet money that 99% of critical systems are not even connected to the internet but air-gapped because, believe it or not, people at those agencies are not that stupid.
Yeah, but Foundry is so ahead, not seeing DataSphere competing there honestly. The only reason is, you already are on SAP and don't want a second system.
Also the engineering / product culture @Palantir is diametrically opposed to what exists at SAP, so I favour Palantir.
You were talking about data engineering. If you do not write tests as a data engineer what are you doing then? Just hoping that you don't fuck up editing a 1000 > line SQL script?
If you use Athena you still have to worry about shuffling and joining, it is just hidden.. It is Trino / Presto under the hood and if you click explain you can see the execution plan, which is essentially the same as looking into the SparkUI.
Who cares about JVM versions nowadays? No one is hosting Spark themselves.
Literally every tool now supports DataFrame AND SQL APIs and to me there is no reason to pick up SQL if you are familiar with a little bit of Python
Still don't get why one of the biggest player in the space, Databricks is overinvesting in Spark. For startups, Polars or DuckDB are completely sufficient. Other companies like Palantir already support bring your own compute.