At StarTree, we've taken a novel approach to querying Iceberg tables: applying Apache Pinot-style indexes to improve performance, lower costs, and increase concurrency… without moving data into a separate system.
In our tests on a ~1TB dataset, this reduced query latency significantly:
* 500+ QPS with sub-second latency
* Complex aggregations <650ms
* ~5–37x faster than ClickHouse and ~4–17x faster than Trino in the same setup
We welcome comments and input from this community!
I ride BART and Caltrain multiple times a week. Trains are delightful. And so are planes.
It typically costs about $100, and takes an hour to fly from SF to LA. The train will be slower and almost certainly more expensive. Where is the benefit?
Long distance trains work in Europe because they are supported by a rich network of public transit options stemming from city centers, and a population that uses transit frequently.
But California public transit is a mess. When you rock up at LA Union Central and want to get out to say, Newport Beach, it's another three hours of navigating buses and the metrolink.
The money would have been far better spent integrating agencies and building out high speed rail in our urban areas. BART should be running out to Sacto, Stockton, Silicon Valley, and Marin. Requisition the Amtrak line and build a highspeed rail from San Diego to Santa Barbara - now that'd be something.
You need to look at use-case alignment as well as performance.
Apache Pinot, Druid and Clickhouse are designed for low-latency analytical queries at high concurrency with continuous ingestion. Pinot is popular because of it's native integration with streaming systems like Kafka, varied indexing, and it's ability to scale efficiently. They're widely used in observability and user-facing analytics – which are how “real-time analytics databases” are commonly perceived today.
Exasol (and SingleStore, Snowflake, BigQuery, etc) are more focused on enterprise BI and complex SQL analytics rather than application serving, or ultra-high ingest workloads. It performs well for structured analytical queries and joins, but it’s less commonly deployed with the user-facing analytics or high volume usage.
Interesting post from Rajat Rana on Meesho's technology blog. Meesho is a huge Indian e-commerce platform. He shares how, and why, they rebuilt their real-time analytics platform on Apache Pinot to support a wider variety of sophisticated features — such as identity merging, deep insights, funnel analysis, and complex user journeys.
(I work at StarTree - a real-time analytics platform powered by Pinot - we helped them implement this)
In our tests on a ~1TB dataset, this reduced query latency significantly:
* 500+ QPS with sub-second latency * Complex aggregations <650ms * ~5–37x faster than ClickHouse and ~4–17x faster than Trino in the same setup
We welcome comments and input from this community!