HackerTrans
TopNewTrendsCommentsPastAskShowJobs

fractaloop

no profile record

comments

fractaloop
·2 anni fa·discuss
AFAIK, it’s limited to fast-forward merge strategies, but you can also create or replace branches and tags, along with cherry-picking snapshots. Additional information can be found in:

• the branching and tagging DDL (https://iceberg.apache.org/docs/latest/spark-ddl/#branching-...)

• Iceberg Procedures (https://iceberg.apache.org/docs/latest/spark-procedures/)
fractaloop
·2 anni fa·discuss
Iceberg (https://iceberg.apache.org) is an open source alternative to Delta Lake that I cannot recommend enough. It organizes your Parquet files (or other serialization formats) in a logical structure with snapshots to allow time travel and git-like semantics for data management and Write-Audit-Publish strategies. My favorite use recently is the idempotent change data capture to ease replication in the event of failures. When your publishing job fails, you can simply replay the same diff between two snapshots and pick up where you left off.
fractaloop
·3 anni fa·discuss
Perhaps a bit too distant, but the Culture series by Iain M. Banks takes place in a utopia built and managed by hyper intelligent benevolent artificial intelligences.