PranaDB – A distributed streaming database, designed to be horizontally scalable(github.com)
github.com
PranaDB – A distributed streaming database, designed to be horizontally scalable
https://github.com/cashapp/pranadb
22 comments
This way of thinking is the future.. and it should IMO also be the future of transactional business logic ("backend dev") programming.
If there was a DB that got popular that focused on the following usecase and covered it really well I think half of the worlds backend programmers would soon be without a job due to the increased efficiency in the sector:
- Focus on persisting insert-only events and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
- Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
Our team programs in this style already with a traditional SQL, it is doable but one does notice one is working against the grain a bit ("if the DB engine only had this thing implemented..."). Problem is all popular DBs started out in the opposite end: That you have mutable "objects" or "documents" or "records". That is IMO a worse way to program for nearly everything.
Programming language trends have shifted from mutable and object oriented to functional and immutable. This DB is an example of the same shift for async work. But we need the same shift for transactional and consistent ACID usecases too.
I recommend the paper "Out of the tar pit" for similar ideas (though without much focus on events as such).
(PS: Yes I know about GDPR and need to delete data. The above is not an exhaustive list of features, but a description about the primary workflow of the intended audience of the DB.)
If there was a DB that got popular that focused on the following usecase and covered it really well I think half of the worlds backend programmers would soon be without a job due to the increased efficiency in the sector:
- Focus on persisting insert-only events and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
- Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
Our team programs in this style already with a traditional SQL, it is doable but one does notice one is working against the grain a bit ("if the DB engine only had this thing implemented..."). Problem is all popular DBs started out in the opposite end: That you have mutable "objects" or "documents" or "records". That is IMO a worse way to program for nearly everything.
Programming language trends have shifted from mutable and object oriented to functional and immutable. This DB is an example of the same shift for async work. But we need the same shift for transactional and consistent ACID usecases too.
I recommend the paper "Out of the tar pit" for similar ideas (though without much focus on events as such).
(PS: Yes I know about GDPR and need to delete data. The above is not an exhaustive list of features, but a description about the primary workflow of the intended audience of the DB.)
Hi, I'm working on something maybe relevant and I'd like to know more,
- Focus on persisting insert-only events
What do you mean? (Are there other events you don't want persisting? why? and what does 'insert-only' mean?)
> and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
Can you explain what this means in practice?
> - Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
This is really unclear - what is 'current state of DB' you're interested in, what is 'allowed' here (and why)
I'd really appreciate some clarification! TIA
- Focus on persisting insert-only events
What do you mean? (Are there other events you don't want persisting? why? and what does 'insert-only' mean?)
> and defining aggregates / materialized views (event sourcing); with all the power of SQL for queries.
Can you explain what this means in practice?
> - Allow DB transactions where the current state of DB is queried (to check if the operation is allowed) and if it is, insert the event.
This is really unclear - what is 'current state of DB' you're interested in, what is 'allowed' here (and why)
I'd really appreciate some clarification! TIA
Insert-only: Sorry bad wording. "Focus on inserting events (which boils down to only inserting into tables, never updating existing rows)"
A database can of course never know that it is "events" you are inserting.
--
Aggregates/materialized views:
(Contrived example warning)
Say that you have a User entity in your system. The Name of the user is only set through two events, UserChangedName or CustomerServiceChangedUserData.
The "aggregate" is simply the calculation of the user's name (and other data) from such events. Declaring this should be done using some variant of functional programming; SQL views, reactive programming, etc
At least in the DB we use, we can declare a SQL view that fits the bill and does such a transform. So User is a view on the events that affects it. But it can not be materialized if it is too complex (materialized = updated when the insert of events happens, so that one does not need to query all the events every time one reads User).
One can do it using triggers, but that is an imperative instead of declarative programming style with all the problems and lack of guarantees.
--
Operation allowed etc:
Say that there is a limit of 5 name changes. However a TooManyNameChangesReviewedByCustomerSupport can be inserted and then you get 5 more.
So user wants to change name. You need to read NameChange events and TooManyNameChangesReviewedByCustomerSupport events, compute whether a change is allowed or not, and only save the event if it is allowed to happen (at which point it becomes the event -- this DB is the source of the event).
What you need now is a transaction that allows you to avoid race conditions. Otherwise a parallel thread could be doing the same thing at the same tim, and you end up jumping from 4 to 6 name changes, when 5 should be the limit.
SQL DBs focus on providing transactional guarantees for UPDATEs, but not so much on guarantees for "only INSERT this row if these other rows in another table are NOT present/has a count less than --" ..
So you end up having to update an aggregate manually in the SQL transaction to avoid races => this is what should be built into the SQL server. (It would just be syntax candy in a sense, saving one from writing code to update the aggregate explicitly, but pretty important syntax candy as it changes the mindset of the developer in an important way)
A database can of course never know that it is "events" you are inserting.
--
Aggregates/materialized views:
(Contrived example warning)
Say that you have a User entity in your system. The Name of the user is only set through two events, UserChangedName or CustomerServiceChangedUserData.
The "aggregate" is simply the calculation of the user's name (and other data) from such events. Declaring this should be done using some variant of functional programming; SQL views, reactive programming, etc
At least in the DB we use, we can declare a SQL view that fits the bill and does such a transform. So User is a view on the events that affects it. But it can not be materialized if it is too complex (materialized = updated when the insert of events happens, so that one does not need to query all the events every time one reads User).
One can do it using triggers, but that is an imperative instead of declarative programming style with all the problems and lack of guarantees.
--
Operation allowed etc:
Say that there is a limit of 5 name changes. However a TooManyNameChangesReviewedByCustomerSupport can be inserted and then you get 5 more.
So user wants to change name. You need to read NameChange events and TooManyNameChangesReviewedByCustomerSupport events, compute whether a change is allowed or not, and only save the event if it is allowed to happen (at which point it becomes the event -- this DB is the source of the event).
What you need now is a transaction that allows you to avoid race conditions. Otherwise a parallel thread could be doing the same thing at the same tim, and you end up jumping from 4 to 6 name changes, when 5 should be the limit.
SQL DBs focus on providing transactional guarantees for UPDATEs, but not so much on guarantees for "only INSERT this row if these other rows in another table are NOT present/has a count less than --" ..
So you end up having to update an aggregate manually in the SQL transaction to avoid races => this is what should be built into the SQL server. (It would just be syntax candy in a sense, saving one from writing code to update the aggregate explicitly, but pretty important syntax candy as it changes the mindset of the developer in an important way)
Very interesting. I know exactly what you mean, of how features chang the mindset of the user.
I'm nowhere near doing what you want but I've been looking for generally useful DB constraints that currently have to be done via triggers in SQL for future work (your syntax candy, basically). Have you an email contact? (I don't do linkedin, I'm very luddite). If not interested just ignore this. Thanks
(Also been wondering about how automatically-maintained-summary aggregations might be needed for businesss, and how to do them - gaian, this is very much future stuff though)
I'm nowhere near doing what you want but I've been looking for generally useful DB constraints that currently have to be done via triggers in SQL for future work (your syntax candy, basically). Have you an email contact? (I don't do linkedin, I'm very luddite). If not interested just ignore this. Thanks
(Also been wondering about how automatically-maintained-summary aggregations might be needed for businesss, and how to do them - gaian, this is very much future stuff though)
This is quite interesting, at $work we use kafka and postgresql / other transactional databases. And we are probably over a man-millennia of work invested in making data consistent between the two. Given a lot of that is based on mindless "REST over Kafka"
What are the biggest mistakes to avoid?
I am itching to start piping around change data capture Debezium logs so tell me now before I inflict a horror on my co-workers.
I am itching to start piping around change data capture Debezium logs so tell me now before I inflict a horror on my co-workers.
Outbox pattern is nice, inbox is more questionable. Depending on size of org you might want to solve that exactly once.
Debezium should be fine as you ingest wal logs to kafka.
Most problems derive when you push data both to kafka and postgres and assume that they are going to be the same. They won’t.
>man-millennia
that's a new one
that's a new one
So how does this differ from https://ksqldb.io/?
I’m looking for something that is stateless itself and stores data in S3 or something. We have a few topics that would be nice to explore and get stats on without having to implement our own query language.
I’m looking for something that is stateless itself and stores data in S3 or something. We have a few topics that would be nice to explore and get stats on without having to implement our own query language.
One massive difference: https://github.com/confluentinc/ksql/blob/master/LICENSE
https://materialize.com/about/
Seems like they could’ve used that instead (plus a Kafka connector) and benefited from a lot of deep research into how to do this optimally
Seems like they could’ve used that instead (plus a Kafka connector) and benefited from a lot of deep research into how to do this optimally
Materialize also isn't open source (just source available) so there could have been strategic reason not to use it. (This is now the first open-source streaming database of its kind [that I've heard of].)
Just had a read through the docs, Materialize is all in-memory. PranaDB persists it to disk.
Could someone give their perspective on why a lot of these companies (big tech) develop their own databases? I don’t want to sound dismissive I am generally curious. What sets this db apart from other streaming services? It must cost a lot to develop something like this?
Nonetheless cool product, thanks for sharing and making it open source.
Nonetheless cool product, thanks for sharing and making it open source.
Big tech companies operate at the edges of the problem space. As a theoretical example, if you need to ingest and process 10TB of numeric data every day, you can probably find an existing DBMS that you can tune to do this. But what if you foresee your volumes growing to 100TB? No one has needed that before.
Patching an existing DBMS means your changes have to mesh well with all its other features. And since you have a lot of talent on payroll already, it's probably easier to write a DBMS that solves your narrow use case, but solves it 100%.
Patching an existing DBMS means your changes have to mesh well with all its other features. And since you have a lot of talent on payroll already, it's probably easier to write a DBMS that solves your narrow use case, but solves it 100%.
The first question they should be asking is, how much of this can I discard immediately, and how much can I discard after storing it in summarised form, then get technical.
Given the Jepson article on the front page and a database announcement, any plans to either Jepson this or consult with Kyle to have it done?
I like the name
Instead of working on making the problem solvable with an exsisting tool.
Personally, I think it would be cool to write my own database as a side project. I would learn a lot. It is a cool thing to work on.
It is also -hard-. Really -hard- to do it. There are millions of odd edge cases that preexisting mature solid database systems have implemented over time as they became aware of them.
Usually, the shortcuts implemented in geen field new databases to make them fast or easy to use end up being impediments laster on down the line.
All of that said in context of being a database being developed by a business to run their own business on. it is, or it will become a giant time consumer. Or you can redesign the code of the actual product to fit the quirks of the homegrown DB.
At that point redesiging the code to fit an existing product will seem like a pretty good idea but now you have sunken cost syndrome.
With all that said, I value diversity. Sometimes something brilliant can come out of doing things that are a bit crazy.
I do wish that a lot more operting systems were created. It could be argued that when you might build a database to solve a product domain, a custom operating system would also get you part of the way.
Most people when they hear "oh you should create your own operating system for that" will tell you, you are nuts. (for good reason). i think the same reaction some come with "You should create your own datasbaase system"