Very cool and well executed project. Love the sprinkle of Rust in all the other companion projects as well :)
The ROAPI(https://github.com/roapi/roapi) project I built also happened to support a similar feature set, i.e. to expose sqlite through a variety of remote query interfaces including pg wire protocols, rest apis and graphqls.
Haha, yeah, we should definitely put a little bit more efforts into SEO :) Everyone is so focused on the hard-core engineering at the moment. I think Matthew from the community is actually working on a new comprehensive benchmark for us at the moment, which I hope will be published soon.
Datafusion out performs spark by a large margin. It is on par with photon based on my experiences, see benchmarks at https://github.com/blaze-init/blaze.
Yes, I designed the code base so that the core of the IO and query logic are abstracted into a Rust library called columnq. My plan is to wrap it with pyo3 so the full API can be accessed as a Python package! If you are interested in helping with this, please feel free to submit a PR. The core library is located at https://github.com/roapi/roapi/tree/main/columnq
It's manual for now, ROAPI is designed for slowly moving datasets. You just hit the data update API to force a refresh.
I have plan to add automated streaming data update in the background, starting with delta lake tables. It should all be very straight forward to implement.
We cache all the data in memory in Arrow format so queries don't need to go through google api, it will only hit google api when a data refresh is needed.
We are hiring new grads across the board if you have proofs for exceptional skills. Feel free to take a look at all our engineering postings at: https://neuralink.com/careers.
There are more companies noticing this now and have stopped asking these questions. For example, we at Neuralink[1] only give out practical programming challenges. If are you good at building practical systems, you should be able to ace our coding interviews without any preparation. No leetcode and no whiteboarding. In fact, I prefer to hire those who doesn't waste time practicing leetcode.
Over the past couple years, I have interviewed at a handful of other startups who also have similar coding interview philosophies.
I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.
In its current form, the main use-case is to load data into memory first then serve them through query apis. Thomas has made some effort to support querying data directly from remote source without loading them into memory: https://github.com/roapi/roapi/pull/71. The underlying query engine, Apache Arrow Datafusion, supports running query on data stream on the granularity of partitions. This is not heavily used in roapi at the moment because I want to nail the in memory serving use-case first.
I looked into Datasette before starting ROAPI. From a product/use-case point of view, to me Datasette focuses more on quick and easy ad-hoc data exploration type of work. ROAPI focuses more production ready online serving of static datasets. So I would expect users to use ROAPI to power micro-services in production with high QPS.
From a technical design point of view, ROAPI authors owns the full stack end to end from query parsing, data format parsing to query execution because I am also a maintainer of Apache arrow and it's sub-project datafusion. The whole project is built with Rust end to end from scratch. Datasette is mostly a wrapper around sqlite. It translates user actions into SQL queries, then execute them on sqlite. In ROAPI, we work at a lower level. We translate REST APIs, GraphQL and SQLs into datafusion logical plans and execute them. Datafusion is also a analytical compute engine optimized for columnar data, so it will be a lot faster for OLAP workload, while sqlite is optimized for OLTP. I also plan to add other type of query capabilities like nearest neighbor vector search for ML applications, etc.
This is true, the core of it is Apache arrow datafusion query engine, which is also a project I help maintain. I doubt you will be able to beat it with PHP though ;) The VM overhead alone will cause a big hit to your performance even if we can get JIT to work.
The ROAPI(https://github.com/roapi/roapi) project I built also happened to support a similar feature set, i.e. to expose sqlite through a variety of remote query interfaces including pg wire protocols, rest apis and graphqls.