Building a Relational Database Using Kafka(yokota.blog)
yokota.blog
Building a Relational Database Using Kafka
https://yokota.blog/2019/09/23/building-a-relational-database-using-kafka/
33 comments
I like the idea, to use Apache Kafka as the underlying log. It's in principle what Martin Kleppmann suggests.
As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL?
I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Furthermore I'd like to expose the API for streaming changes into the Browser or wherever you want to :-)
As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL?
I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Furthermore I'd like to expose the API for streaming changes into the Browser or wherever you want to :-)
I've been itching for a while to build something on top of BookKeeper's DistributedLog. I suggest ditching the Pulsar and just using the DL.
I've already written with one of the core committers to BookKeeper and yes, I think I'll use it.
That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.
I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).
I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)
So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.
BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.
I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).
I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)
So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.
BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it:
https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
> The thing I don't like is that we also need BookKeeper, but yeah.
Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]
> sirixdb
Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.
Wish you success in your effort!
https://jack-vanlightly.com/blog/2018/10/2/understanding-how...
> The thing I don't like is that we also need BookKeeper, but yeah.
Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]
> sirixdb
Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.
Wish you success in your effort!
Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!
To be clear, it is BookKeeper that can alternatively use etcd not Pulsar.
https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...
https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...
https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
Absolutely. Pulsar looks nice, too, but I think BookKeeper not needing ZK gives it an edge over Kafka.
I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"?
From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"?
From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
Pulsar is built on top of BK. BK has it own consensus mechanism (ZK, etcd), and Pulsar also has its own (low traffic) consensus via ZK. The article cited above does a very good job of addressing the advantages of Pulsar's architecture over Kafka's. Pulsar also provides for user defined functions (that can run on the broker). This is reminiscent of Postgres's User Defined Functions.
BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that.
Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that.
Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
Thanks. Right, I know Pulsar is built on top of BK — it's just unclear how much extra value it adds.
I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning.
I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable.
I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant
Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.
I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning.
I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable.
I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant
Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.
Regarding the question if Apache Pulsar or Apache BookKeeper for me it's not so clear which of both to use. I think it adds a lot of value if you add the WAL to the public API. Forgot how it's called, I think change data types (CDTs) without searching on google!? So basically you have the stream of events and expose these for use by applications :-)
> BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties.
With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.
One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties.
With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.
From personal experience, considering the reliability of Kafka, I’d implement this in reverse. Most relational databases are far more resilient in operation.
One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for FoundationDB.
One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for FoundationDB.
Interesting, what were the failure modes you encountered?
This is fetishizing complexity for no reason. Makes for an interesting blog post, but if any company told me they were doing this I would run the other direction immediately.
Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components.
Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.
Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.
Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.
Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.
You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools.
At the same time, OP is right: a business doing this is making poor engineering decisions.
At the same time, OP is right: a business doing this is making poor engineering decisions.
We don't know that yet! The post explains what they did but not the why of the problem. Maybe they have a good reason, maybe they don't.
But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel".
Specifics matter!
But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel".
Specifics matter!
I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples.
I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.
I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.
So question: is there a good reason to use Avro over protobufs?
Just from a hand wavy aesthetic perspective I like protobufs but I'd like to get a clear picture.
Just from a hand wavy aesthetic perspective I like protobufs but I'd like to get a clear picture.
If you are using this for your shopping website I would definitely agree. If you are building a managed database service like DynamoDB or Aurora then complexity may be warranted. Great for companies like Google and Amazon that need something that will work best for them and have the engineering talent to make it stick.
Projects like this are how we get solutions that blow existing stuff out of the water. Although a lot just end up being fun projects that amount to nothing but an entertaining blog article.
Projects like this are how we get solutions that blow existing stuff out of the water. Although a lot just end up being fun projects that amount to nothing but an entertaining blog article.
Agreed. To expand on your point, I believe this is pretty close to exactly what Amazon did to build Aurora, which is probably the most exciting development I've seen as an engineering leader in the past decade. Suddenly, the one crucial, stateful component of my system that could ever present sharding issues at scale became something I could pay a premium to not have to think about (to a point -- believe it's still 64TB for Aurora).
What's extra cool about this for me is how it illustrates to me how far open source components that build on the Apache Kafka ecosystem have come, to the degree that each fairly complex component of relational databases that expose a full RDBMS system from the basic building block of a log have been fully developed (to some production capacity) in the Kafka ecosystem. That's amazing! What does this mean?
It means that you could theoretically (please don't hurt me) implement a relational DB backed by a twitter DB log, or a blockchain, or any other event log that you can't necessarily use traditionally as a WAL. It also introduces a lot of really interesting possibilities for data integration. Definitely has my mind reeling a bit.
What's extra cool about this for me is how it illustrates to me how far open source components that build on the Apache Kafka ecosystem have come, to the degree that each fairly complex component of relational databases that expose a full RDBMS system from the basic building block of a log have been fully developed (to some production capacity) in the Kafka ecosystem. That's amazing! What does this mean?
It means that you could theoretically (please don't hurt me) implement a relational DB backed by a twitter DB log, or a blockchain, or any other event log that you can't necessarily use traditionally as a WAL. It also introduces a lot of really interesting possibilities for data integration. Definitely has my mind reeling a bit.
how else would one implement microservices then?
Do you run away from microservices too?
Do you run away from microservices too?
Interesting article, but I think it is missing a solid comparison of such "overlay SQL" with more traditional database engine.
It isn't enough to show something can be done. One must also think if it should be done.
It isn't enough to show something can be done. One must also think if it should be done.
The author is doing this purely for fun (I think, I'm not the author). They link to this: https://www.confluent.io/product/ksql/ which is apparently a production tested system that provides a SQL interface to a Kafka backend.
> provides a SQL interface to a Kafka backend.
Not sure if thats an accurate description of ksql.
Not sure if thats an accurate description of ksql.
Why would I want a relational database living in Kafka? MySQL or Postgres are great at what they do.
A lot of the low-level behaviour is hard to control, at least in terms of having a well-known public interface. E.g. transaction isolation level is database-global, limited control over when updates to indices happen, limited direct control over MVCC. The internals of MySQL or Postgres look a lot like Kafka, but the event-transforming parts are hidden inside a black box.
I love PG and rdbms in general, but certainly exist SO MANY things that could be improved with RDBMS.
1- RDBMS implement an imperfect view of the relational model. Is like to say Java is the only OO inplemented in the world.
2- SQL is the standard... and is not practical to say to eliminate it... BUT RDBMS are constrained for the subpar language SQL is. Modern additions are nice... but that is to say that postcript is a nice way to do apps.
Also, a much better language could expand the role of RDBMS.
This is not weird. I live this way when working in FoxPro:
- UI on fox lang. - DB on fox lang. - Reports on fox lang. - Scripting on fox lang. - Web on fox lang - Triggers on fox lang - OO on fox lang.
And all that still with clean separation of logic and components.
3- RDBMS were made by a certain mindset and use case of the 80s. Still so good that can be valid today, but it could get extended. JSON support is just a tiny example of that.
4- Why you can't do "SELECT .. FROM index"? Why you need to create a table to get an index or FTS? Is weird limitation.
The relational model is fine too for KV stores.
----
"Relational database" is a database made on top of the relational MODEL. You could get MANY IMPLEMENTATIONS of that with different use cases. Similar how the functional MODEL and the OO MODEL is not frozen with a single language... that is not that good for app development.
With a better lang, you could eliminate massive ORMs in a nice way!
P.D: Remember, this IS NOT teory. Is proven. This is how was with the dbase family of langs!
1- RDBMS implement an imperfect view of the relational model. Is like to say Java is the only OO inplemented in the world.
2- SQL is the standard... and is not practical to say to eliminate it... BUT RDBMS are constrained for the subpar language SQL is. Modern additions are nice... but that is to say that postcript is a nice way to do apps.
Also, a much better language could expand the role of RDBMS.
This is not weird. I live this way when working in FoxPro:
- UI on fox lang. - DB on fox lang. - Reports on fox lang. - Scripting on fox lang. - Web on fox lang - Triggers on fox lang - OO on fox lang.
And all that still with clean separation of logic and components.
3- RDBMS were made by a certain mindset and use case of the 80s. Still so good that can be valid today, but it could get extended. JSON support is just a tiny example of that.
4- Why you can't do "SELECT .. FROM index"? Why you need to create a table to get an index or FTS? Is weird limitation.
The relational model is fine too for KV stores.
----
"Relational database" is a database made on top of the relational MODEL. You could get MANY IMPLEMENTATIONS of that with different use cases. Similar how the functional MODEL and the OO MODEL is not frozen with a single language... that is not that good for app development.
With a better lang, you could eliminate massive ORMs in a nice way!
P.D: Remember, this IS NOT teory. Is proven. This is how was with the dbase family of langs!
I came here to ask this question. It's a really solid technical article. But apart from the sake of doing it, why would anyone ever want to? I'm not suggesting there's no scenario in which one would want to build a SQL database using Kafka, I'm just asking if anybody knows of one.
Why would I want DOOM running on a 148x64 LCD Canon printer screen?
Because I can - https://www.youtube.com/watch?v=NPWi5yJK3zo
Because I can - https://www.youtube.com/watch?v=NPWi5yJK3zo
I don't see the article claiming people should drop PostgreSQL and use this, it's just asking people to give it a try if they are interested.
Much more interesting would be to discuss what capabilities would be different if this were a mature product.
For instance, this sounds interesting:
> One advantage of using Kafka is that multiple servers can all “tail” the same set of topics. This allows multiple KarelDB servers to run as a cluster, with no single-point of failure. In this case, one of the servers will be elected as the leader while the others will be followers (or replicas). When a follower receives a JDBC request, it will use the Avatica JDBC driver to forward the JDBC request to the leader. If the leader fails, one of the followers will be elected as a new leader.