Show HN: Apache Iceberg FDW for Postgres(fdw.dev)

1 pointsby kiwicopple8 months ago1 comments

Show HN: Apache Iceberg FDW for Postgres

http://fdw.dev/catalog/iceberg/

This is a Postgres Foreign Data Wrapper for Postgres, which you can use to query Apache Iceberg. A FDW allows you to access and query data stored in external data sources as if they were native Postgres tables.

It is compatible with the Iceberg REST catalog and S3 Tables (AWS S3's latest offering). Currently it supports SELECT and INSERT operations.

You can map an entire Iceberg namespace into your Postgres database like this:

    import foreign schema "warehouse"
    from server iceberg_server 
    into "my_iceberg_warehouse";

And then query them using a standard SELECT query from your Postgres database:

    select * from warehouse.my_table where id = 42;

You can also create tables from within your Postgres database which will then be created inside Iceberg using the `create_table_if_not_exists` option:

    create foreign table new_table (
      id bigint,
      name text
    )
    server iceberg_server
    options (
      table 'warehouse.new_table',
      rowid_column 'id',
      create_table_if_not_exists 'true'
    );

The FDW extension used PGRX + the iceberg-rust libraries. The iceberg ecosystem is still very nascent, but we're excited about some architectural patterns it enables. We'll be adding a few resources to the ecosystem, including some libraries that make it easier to use. You can install this FDW on any self-hosted Postgres database and it's available today on the supabase platform.

1 comments

kiwicopple8 months ago

GitHub repo is here if you want to use this for self-hosted Postgres databases:

https://github.com/supabase/wrappers