Dozer: Fast API Data Serving from Multiple Sources with Real-Time Transformation(getdozer.io)

2 pointsby snork_alt3 yıl önce1 comments

Dozer: Fast API Data Serving from Multiple Sources with Real-Time Transformation

https://getdozer.io/blog/like-airbyte-elasticsearch-15x-faster/

1 comments

> I decided to experiment with this setup and the NY Taxi Dataset. The initial goal was to populate ElasticSearch with ~14 million rows, loading data from a compressed parquet file of ~350 MB.

> I tried multiple times, but the operation failed continuously, due to JVM memory constraints

Here's a script https://github.com/db-benchmarks/db-benchmarks/blob/main/tes... which loads 1.7B NYC taxi ride documents into Elasticsearch.