TL;DR: POST JSON to an HTTP endpoint. Get Parquet and compressed segments in your S3 bucket. No Kafka, no Kinesis, no glue code.
EdgeMQ ingests newline delimited JSON from one or more global endpoints (dedicated vm's). Data is delivered to your S3 with commit markers in one or more formats of your choosing:
* Compressed WAL segments (.wal.zst) for replay i.e. raw bronze
* Raw/opaque Parquet (keeps the original payload in a payload column + ingest metadata).
* Schema-aware Parquet - materialized views defined in YAML
Feedback request:
The platform is now open for public beta (there are a number of current live users) and I am keen to collect further feedback and explore use cases.
I would be grateful for comments and thoughts on:
* Use cases - are there specific ingest use cases that you use regularly?
* Ingest formats - the platform supports NDJSON - do you use others?
* output formats - are there other transformations outside of the 3 supported that would be useful?
* Output locations - S3 is supported today, but are there other storage locations that would simplify your workflows? Object store has been the target to date.
0 comments
—
Show HN: Edge HTTP to S3, now with schema-aware Parquet · HackerTrans
TL;DR: POST JSON to an HTTP endpoint. Get Parquet and compressed segments in your S3 bucket. No Kafka, no Kinesis, no glue code.
EdgeMQ ingests newline delimited JSON from one or more global endpoints (dedicated vm's). Data is delivered to your S3 with commit markers in one or more formats of your choosing: * Compressed WAL segments (.wal.zst) for replay i.e. raw bronze * Raw/opaque Parquet (keeps the original payload in a payload column + ingest metadata). * Schema-aware Parquet - materialized views defined in YAML
Feedback request:
The platform is now open for public beta (there are a number of current live users) and I am keen to collect further feedback and explore use cases.
I would be grateful for comments and thoughts on: * Use cases - are there specific ingest use cases that you use regularly? * Ingest formats - the platform supports NDJSON - do you use others? * output formats - are there other transformations outside of the 3 supported that would be useful? * Output locations - S3 is supported today, but are there other storage locations that would simplify your workflows? Object store has been the target to date.