HackerTrans
TopNewTrendsCommentsPastAskShowJobs

mattforrest

no profile record

Submissions

Price-performance benchmarks for spatial SQL (Databricks Serverless vs. Sedona)

sedona.apache.org
3 points·by mattforrest·6 maanden geleden·1 comments

Apache Sedona tutorial for Spark based spatial processing [video]

youtube.com
2 points·by mattforrest·10 maanden geleden·1 comments

comments

mattforrest
·5 maanden geleden·discuss
I wrote a post about using H3 or any DGGS for that matter. Yes it speeds things up but you loose accuracy. If search is the primary concern it can help but if any level of accuracy matters I would just use a better engine with GeoParquet to handle it. https://sedona.apache.org/latest/blog/2025/09/05/should-you-...
mattforrest
·6 maanden geleden·discuss
SpatialBench is an open benchmark suite for spatial SQL. The goal is to compare engines on price-performance, since serverless makes raw “x times faster” claims hard to interpret.

I used it to compare Databricks SQL Serverless (Medium) vs Databricks Jobs clusters with Apache Sedona 1.7 across 12 queries (from simple filters to joins, distance joins, multi-way joins, KNN) at SF100 and SF1000 (SF1000 is roughly 500GB uncompressed Parquet).

TLDR apart from one query, Sedona was up to ~6x better on cost per query, and also covered more queries under the same 10 hour timeout guardrails. Some queries didn’t finish or errored on either side, so there is a capability matrix in the post.
mattforrest
·10 maanden geleden·discuss
I wrote a book on PostGIS and used it for years and these single node analytical tools make sense when PostGIS performance starts to break down. For many tasks PostGIS works great, but again you are limited by the fact that your tables have to live in the DB and can only scale as much as the computing resources you have allocated.

In terms of number of functions PostGIS is still the leader, but for analytical functions (spatial relationships, distances, etc) having those in place in these systems is important. DuckDB started this but this has a spatial focused engine. You can use the two together, PostGIS for transactional processing and queries, and then SedonaDB for processing and data prep.

A combination of tools makes a lot of sense here especially as the data starts to grow.
mattforrest
·10 maanden geleden·discuss
I put together a tutorial for Apache Sedona which brings geospatial to Spark.

A project that crunches real estate and satellite imagery data with scalable spatial joins

Sedona basically makes spatial at scale way more accessible. Instead of rolling your own hacks, you get Sparks distributed compute with geospatial APIs baked in.
mattforrest
·vorig jaar·discuss
They aren’t reliable correct actually. The boundaries that the Census publishes are called Zip Code Tabulation Areas which are approximations of zip codes and include overlaps.
mattforrest
·vorig jaar·discuss
Just use a spatial query. That’s what they are made for.
mattforrest
·vorig jaar·discuss
Yeah but we have that already in the census hierarchy. Plus you have to pay to access Zip+4 geospatial data and it changes sometime as frequently as quarterly
mattforrest
·vorig jaar·discuss
Well put
mattforrest
·vorig jaar·discuss
Yeah but Zip+4 represent a collection of houses not a polygon so not useful for aggregations or statistical work
mattforrest
·vorig jaar·discuss
Not necessarily true. The population isn't balanced at all between many. Census units are.
mattforrest
·vorig jaar·discuss
Well you hit on all the points that discuss the compromises that zip codes offer. Just because you have them in your data doesn't mean that they can produce anything useful. You are correct that no one knows their census unit is (if you are thinking from someone entering this on a website) but collecting location or address will be a lot better.

Fact is a lot of web data contains a zip but if you can collect something better it will usually render better results. Unless you are analyzing shipments then that is fine.
mattforrest
·vorig jaar·discuss
Funny to see this one pop up today (I wrote this one way back when) but I just refreshed it into a video on my channel: https://www.youtube.com/watch?v=x-opv4REEic