1 - I think if we were sticking with the JVM, I do wonder if Lucene would be the right choice in that case
2 - It's a great tool with a lot of tuneability and support!
3 - We've been using it for K8s logs and OTEL (with Jaeger). Seems good so far, though I do wonder how the future of this will play out with the $DDOG acquisition.
These are great projects, we use DuckDB to inspect our data lake and for quick munging.
We will have some more blog posts in the future describing different parts of the system in more detail. We were worried too much density in a single post would make it hard to read.
It's a bit difficult at the moment, given we have a lot of proprietary data at the moment and a lot of the logic follows it. I'm hoping we can get it to a state where it can be indexed and serving OSM data but that is going to take some time.
That being said, we are currently working on getting our Google S2 Rust bindings open-sourced. This is a geo-hashing library that makes it very easy to write a reverse geocoder, even from a point-in-polygon or polygon-intersection perspective.
Author here! We were really motivated to turn a "distributed system" problem into a "monolithic system" from an operations perspective and felt this was achievable with current hardware, which is why we went with in-process, embedded storage systems like RocksDB and Tantivy.
Memory-mapping lets us get pretty far, even with global coverage. We are always able to add more RAM, especially since we're running in the cloud.
Backfills and data updates are also trivial and can be performed in an "immutable" way without having to reason about what's currently in ES/Mongo, we just re-index everything with the same binary in a separate node and ship the final assets to S3.
Radar Labs | Technical PMs & Software Engineers (infra, platform, data, full-stack, mobile) | Remote US / New York City | Full Time
https://radar.com Radar Labs is a location infrastructure SaaS platform.
We've recently raised our Series C and are a small, but mighty engineering team (9 engineers!). We're based across the US, with an HQ in NYC in Union Square.
There's a ton of opportunity to make an impact and work on a variety of things, as we're a pretty fluid team:
- Infrastructure (e.g. Evaluate our production workloads on ARM64 and build out our AWS stack with Terraform CDK)
- Platform (e.g. Build out abstractions for our product engineers to ingest and compute geo queries at high-scale)
- Data (e.g. Process, deduplicate, and evaluate 10s of millions of POIs from a variety of vendors and open data)
- Mobile (e.g. building a control system that automatically configures optimizing battery-life and location accuracy for different use-cases over time)
- Product (e.g. build tools to visualize and debug location data at scale)
We power location experiences for over 100M devices, serving over 10k QPS of geo queries. The numbers are growing day-by-day!
Radar Labs | Software Engineer (infra, platform, data, full-stack, mobile) | Remote US / New York City | Full Time
https://radar.com Radar Labs is a location infrastructure SaaS platform.
We've recently raised our Series C and are a small, but mighty engineering team (9 engineers!). We're based across the US, with an HQ in NYC in Union Square.
There's a ton of opportunity to make an impact and work on a variety of things, as we're a pretty fluid team:
- Infrastructure (e.g. Evaluate our production workloads on ARM64 and build out our AWS stack with Terraform CDK)
- Platform (e.g. Build out abstractions for our product engineers to ingest and compute geo queries at high-scale)
- Data (e.g. Process, deduplicate, and evaluate 10s of millions of POIs from a variety of vendors and open data)
- Mobile (e.g. building a control system that automatically configures optimizing battery-life and location accuracy for different use-cases over time)
- Product (e.g. build tools to visualize and debug location data at scale)
We power location experiences for over 100M devices, serving over 10k QPS of geo queries. The numbers are growing day-by-day!
Radar Labs | Software Engineer (infra, platform, data, full-stack, mobile) | Remote US (CA/EU soon!), New York City | Full Time
https://radar.com Radar Labs is a location infrastructure SaaS platform.
We've recently raised our Series C and are a small, but mighty engineering team (9 engineers!). We're based across the US, with an HQ in NYC in Union Square.
There's a ton of opportunity to make an impact and work on a variety of things, as we're a pretty fluid team:
- Infrastructure (e.g. Evaluate our production workloads on ARM64 and build out our AWS stack with Terraform CDK)
- Platform (e.g. Build out abstractions for our product engineers to ingest and compute geo queries at high-scale)
- Data (e.g. Process, deduplicate, and evaluate 10s of millions of POIs from a variety of vendors and open data)
- Mobile (e.g. building a control system that automatically configures optimizing battery-life and location accuracy for different use-cases over time)
- Product (e.g. build tools to visualize and debug location data at scale)
We power location experiences for over 100M devices, serving over 10k QPS of geo queries. The numbers are growing day-by-day!