jeadie·hace 3 meses·discussThis is exactly what we found. Ingest rates were tough. We partitioned and ran over multiple duckdb instances too (and wrangled the complexity).We ending up building a Sqlite + vortex file alternative for our use case: https://spice.ai/blog/introducing-spice-cayenne-data-acceler...
jeadie·hace 3 meses·discussYou might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest
jeadie·hace 10 meses·discussWe’re building vector indexes into Datafusion for search (starting with S3 vectors).Open source at https://github.com/spiceai/spiceai
We ending up building a Sqlite + vortex file alternative for our use case: https://spice.ai/blog/introducing-spice-cayenne-data-acceler...