To anyone that has use for this, I also recommend trying the fish shell. It takes some getting used to at first but the sheer time save I realize with just autocomplete is monumental!
Performance numbers are coming, please look out for them in a future blog post.
Concrete details about the architecture included in https://rockset.com/Rockset_Concepts_Design_Architecture.pdf.
If you think we could add value to you beyond what you get with Aurora PG, then I'd welcome a chance for an evaluation on a 100% free trial (no credit card required) where you could test out the performance for yourself with your own data.
Totally respect the skepticism. please reach me directly at anirudh at rockset.com if you'd like to chat further.
You are right in that it uses SSD and indexes under the hood to power the SQL queries. There are volume discounts available at higher storage volumes in the 10s of TB range, so, the cost wouldn't scale linearly.
Rockset builds multiple indexes to enable low latency SQL queries that can be served directly into applications. This might not make sense for say - storing a lot of log data and querying it rarely, and makes more sense for data that needs to be actively queried with low latency.
As the data grows, we manage shards under the hood to ensure that the data spread across more nodes which in turn helps us use more parallelism. It is possible to manage the query performance and the data size independently in our architecture. Look out for numbers in a future blog post.
That's just for streaming input by default. We do work with users and can increase it if the use-case demands it. For bulk ingest from sources like S3, that limit does not apply and it typically does many MB/s.
In terms of use-cases, we see things differently from a product like Athena. We are focused on ETL-free real-time analytics and applications which make use of the fact that we construct multiple indexes automatically behind the scenes to enable low latency and scalable query serving.
Hi yingw787, I work on the product team at Rockset. Thanks for your thoughts!
I'll try and answer your questions below.
- The different file formats get indexed and turn into a Rockset specific format which ensures that irrespective of the file type you get excellent performance for your SQL queries.
This also means you can JOIN data from different sources (containing files in different formats) using SQL irrespective of the source formats.
- Depending on the complexity of the SQL queries, the latency can range from low tens of milliseconds to a few seconds. Since we index ALL the fields in several ways,
if we're able to use our indices to accelerate the query (which is almost always the case), it will likely be in the 10-200 milliseconds range for a wide range of analytical queries.
Look out for some numbers in the future.
- Data cleaning is something we facilitate through the use of our delete/update records API that lets you mutate the index and remove/update the records that you consider to be containing bad data. Since Rockset supports schemaless ingest (https://rockset.com/blog/from-schemaless-ingest-to-smart-sch...), error documents don't really break anything and you can work around them by writing a query that ignores them. We are interested in providing visibility into the data so that you can quickly detect issues with the data and fix them.
I tend to take one-off classes at Stanford to try and see if I can short-circuit some learning that would otherwise come the hard way. Aside from that, sometimes meeting other entrepreneurs, working on long term projects that I don't get to do during the week, and meeting friends to have a few laughs together.