james_cowling

letztes Jahrdiscuss

I'm giving the first talk of the session and while I'll be talking about ideas we've actually built and tested in Convex, the point I want to get across is that I think the industry has gone in the wrong direction with exposing databases to clients and using an inexpressive declarative language to query them. Will be trying to avoid any shilling because I feel like this is a pretty important discussion to have.

james_cowling

vor 2 Jahrendiscuss

Yes the readsets aren't based on lists of objects but rather the data ranges that a query touches, i.e., they don't suffer from phantom read anomalies.

If you were to attempt to fetch all items from an empty shopping cart, that query will automatically be invalidated if any item is added to the cart, even though there were no documents returned from the original query. Query intersection always works.

The aggregate example isn't a great one in Convex since it doesn't currently support built-in aggregates - a full-table aggregate involves a table scan and therefore the readset would be the entire index. We may add built-in aggregates that are incrementally computed if there's enough demand.

james_cowling

vor 2 Jahrendiscuss

Convex is FSL licensed which means you can basically do whatever you want with it re running a service, other than directly competing with Convex, i.e., reselling the Convex platform. There are no limitations on you running a business on top of this and no obligation to pay anything.

After 2 years the code becomes Apache 2.0.

Note that if you're running the open source release you're on the hook for managing and scaling your own reliable infrastructure, as described in the open source readme.

I talk more about the philosophy re open sourcing convex in https://softwareengineeringdaily.com/2024/03/20/going-open-s...

james_cowling

vor 2 Jahrendiscuss

(convex cofounder here)

Convex is in a very comfortable financial position but it is also open source: https://github.com/get-convex/convex-backend

james_cowling

vor 2 Jahrendiscuss

Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time.

Internally the very bottom of the stack on the hosted product is actually just a write ahead log on RDS and in the open source product it's just sqlite. We'll likely eventually build our own durable write ahead log, and have plenty of experience doing so, but this hasn't been needed so far.

james_cowling

vor 3 Jahrendiscuss

Convex cofounder here. Initial focus has been low-latency OLTP database workloads, which is pretty tied to the user-facing end-to-end reactivity angle we're pushing. As a bonus feature we also have built-in strongly consistent full text search, but not vector search.

So far our answer for folks who want alternative storage/query engines is to use our streaming Airbyte source connector or write directly to Pinecone, Snowflake, etc. This should work great for most devs.

There are likely always going to be some developers who want to use a particular third party database in addition to Convex, but we plan to expand built-in support for most workloads over time so that Convex is a truly full-stack backend replacement.

james_cowling

vor 3 Jahrendiscuss

(kmod was the founding engineer on the project and the guy who came up with the name!)

james_cowling

vor 3 Jahrendiscuss

We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then!

(a bunch of the early MP folks work at Convex now)

james_cowling

vor 3 Jahrendiscuss

"It’s fairly easy to design a system with astronomically high durability numbers. 24 nines is a mean time to failure of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe then it might be time to reevaluate your priorities.

Should we trust these numbers though? Of course not, because the secret truth is that adherence to theoretical durability estimates is missing the point. They tell you how likely you are to lose data due to routine disk failure, but routine disk failure is easy to model for and protect against. If you lose data due to routine disk failure you’re probably doing something wrong."

https://medium.com/@jamesacowling/how-many-nines-is-my-stora...

james_cowling

vor 3 Jahrendiscuss

We had a lot of requests for open sourcing MP but I don't think it would have been very useful. The code itself is interesting to look at but the system is designed for very large data sets, has a lot of moving parts to manage and configure, and would be very difficult to operate at anything below at least double-digit petabytes.

MP was designed for multiple exabytes across many data centers and geographic regions. A different system design (or just using S3) would be more appropriate for use at smaller scales.

james_cowling

vor 3 Jahrendiscuss

Yeah we have a few talks about this and a chapter about this very issue in https://www.oreilly.com/library/view/seeking-sre/97814919788.... Totally agree that in a well designed system the sources of data loss are certainly not disk failures.

As far as I know Magic Pocket has had 100% durability, but that's obviously beside the point.

james_cowling

vor 3 Jahrendiscuss

We designed and built the system almost completely ground-up, all the way down to the disk scheduler, but yeah we probably took the term OSD from Ceph.

james_cowling

vor 4 Jahrendiscuss

Yep that’s me! I have my full name on here so wasn’t hiding but good point to be clear about disclosing.

james_cowling

vor 4 Jahrendiscuss

Firebase has been a huge influence but with the rise of serverless there are a new generation of platforms people should be checking out instead, such as Convex, Supabase, etc.

Convex in particular is designed far more for end-to-end consistency rather than just a database to talk to: https://docs.convex.dev/understanding/convex-vs-firebase

comments