Show HN: Typed-arrow – compile‑time Arrow schemas for Rust(github.com)
github.com
Show HN: Typed-arrow – compile‑time Arrow schemas for Rust
https://github.com/tonbo-io/typed-arrow
8 comments
I'm pretty deep into the arrow ecosystem and have been dreaming about something like this for a while now. Nice to see!
I'm also imagining what this would look like in other languages (Java, Go, Python) - you can't do as much at compile time of course, but there are still marginal improvements to be made over the dynamism of the standard arrow libraries, when the use case benefits from it.
I'm also imagining what this would look like in other languages (Java, Go, Python) - you can't do as much at compile time of course, but there are still marginal improvements to be made over the dynamism of the standard arrow libraries, when the use case benefits from it.
I had no idea what Arrow is: https://arrow.apache.org or arrow-rs: https://github.com/apache/arrow-rs
Cool, curious if there is any willingness or interest in adding a geoarrow feature, or generally some form of integration with that crate? (Though to be honest I'm not currently clear up to date on the stability/usability of geoarrow at this current point in time)
Sure I'd love to, could you share your practice or what you are doing on top of Parquet?
My project hasn't really gotten to the level of actually exporting data to any database yet, but basically I'm just using standard openstreetmap data exported to geoparquet format.
That is then combined with a bunch of time series sensor readings combined with a gps data, (I'm basically still working on the device firmware so I don't have any actual data export happening yet)...
But it is a pretty basic combination of timeseries and geospatial data.
That is then combined with a bunch of time series sensor readings combined with a gps data, (I'm basically still working on the device firmware so I don't have any actual data export happening yet)...
But it is a pretty basic combination of timeseries and geospatial data.
How is this different from https://github.com/Swoorup/arrow-convert?
typed-arrow avoids any kinds of runtime side effects: cost, errors, etc.
I congratulate you. This is a clever extension of a great framework.
When working with arrow-rs, we noticed that schemas are declared at runtime. This often leads to runtime errors and makes development less safe.
typed-arrow takes a different approach:
- Schemas are declared at compile time with Rust’s type system.
- This eliminates runtime schema errors.
- And introduces no runtime overhead — everything is checked and generated by the compiler.
If you’ve run into Arrow runtime schema issues, and your schema is stable (not defined or switched at runtime), this project might be useful.