You might enjoy this project, which ties to do basically exactly what you described: stick everything in a database and let it drive the app: https://riffle.systems/essays/prelude/
It’s still very much a research prototype but we should have some more writing out soon.
Makes sense, especially the part about the "recipe" being more about organizational capacity than code.
Do you think it's important for volunteers to live in the region that they're trying to report on? Concretely, could a network based in the Bay Area make reports about availability in Michigan?
I might be able to get a sufficient group in Massachusetts, but the state hasn't approved general availability yet, despite CDC guidance.
You're right the Record Layer doesn't yet have join support, but note PR #306 [1]. We support aggregate indexes but can't run aggregate “reports” that aren't backed by indexes. The rationale for this is covered in the paper (see [2]), but note that doesn't preclude such support being added into or on top of Record Layer (also discussed in the paper).
The Record Layer does a whole bunch of work to deal with index maintenance (see our docs [3])). Our “index maintainer” abstraction (discussed in the paper) makes maintaining our indexes (including those that are basically materialized views) completely seamless from the user's perspective, even for updates and deletes. We also have a lot of tooling for making efficient schema migrations. For example, schema migrations are performed lazily (when the data is accessed), so they aren't limited by the 5 second transaction limit. If you add/remove/change indexes, they'll be put into a “write-only” mode where they'll keep accepting writes while an “online indexer” builds the index over multiple transactions. We even have fancy logic to automatically adjust the size of the transactions if they start failing due to contention or timeouts!
Basically, the Record Layer solves a lot (but not all) of the pain points that shows up when you don't know your access patterns from the beginning. The paper talks a bit about how CloudKit uses some of those features.
(I'm from the iCloud team that works on the Record Layer.) Both building a relational database and implementing a proper SQL interface on top of it are huge projects. The SQL spec is large and complicated, so achieving true compatibility (as opposed to superficial compatibility) is challenging. Even worse, once you have a SQL interface users expect to be able to throw any SQL that they give to, say, Postgres, and have it work just as well, which requires a ton of detailed work on the query optimizer.
The client/server distinction isn't terribly strong in the FDB world. The FDB client is unusual in that it's a (stateless) part of the FDB cluster itself. You could therefore embed it in the client itself or build an RPC service around it. The Record Layer takes the same approach---it's just a Java library---so you could either embed it in the client application or build some kind of wire protocol for accessing it. One could have an embedded SQL layer like SQLite or H2 with no additional server beyond the cluster or a separate SQL layer network server that acted more like Postgres or MySQL.
The Record Layer was designed for use cases that don't need a SQL interface, so we focused on building the layer itself. That said, the Record Layer exposes a ton of extension points so there's a fluid boundary between what needs to live in its main codebase and what can be implemented on top. There are almost certainly enough extension points to implement a SQL interface as another layer on top of the Record Layer. For example, you could add totally new types of indexes outside of the Record Layer's codebase, if that were needed for SQL support. It's still a lot of work, especially on the query optimizer. Perhaps the community is up to that challenge. :-)
I'm a friend and colleague of Michael's at MIT. This article is very nice and gives a good summary of what made Michael so special. In a lot of ways, Michael was the animating "spirit" of the MIT theory group. He had an encyclopedic knowledge and incredibly deep understanding of basically every area of computer science, and many areas beyond; in the short year that I knew him, we had conversations about everything from convex optimization to computer architecture to rent control laws to Medieval philosophy.
Michael was a truly remarkable researcher. Ludwig's comments about him being the type that you "only see a couple of times in a generation" are accurate. I also recommend watching the start of Yin-Tat Lee's recent talk [1] at the Simons Institute. Yin-Tat is a prolific researcher himself, so his comments carry a lot of weight.
For those wondering about Michael's publication count: computer science (and especially theoretical computer science) is a "high publication" field, in part because of the nature of publishing in conferences and in part because the field is young and there are many good open problems. Still, Michael's publication record is abnormally strong and reflects his collaborative nature. Regarding the comments about co-authorship, Michael could easily have been a co-author on a dozen more papers if he had cared, since he often contributed the main ideas to projects that he never formally joined. This was definitely my experience collaborating with him. I expect that Michael will be more prolific in the next year than many living researchers, from the point of view of publishing.
His papers (incomplete list here [2]) are very well written, by the way. I recommend checking them out.
The most incredible thing about Michael was the way he learned. If you talked about something that he didn't understand, he'd quiz you about it until he did. And he did this with everyone, from brand new grad students like me to famous professors.
At the same time, Michael was incredibly generous. He liked to talk, and you could interrupt him at any time and he'd explain everything to you with astounding patience. Michael wasn't in science for glory; he just really loved learning and teaching. He's already profoundly missed and our entire community is shocked by his untimely passing. My deepest condolences go to Michael's family.
We hope to have a memorial website up soon, especially since Michael was too humble to have much of an online presence.