Starbase: SQLite on the Edge(starbasedb.com)
starbasedb.com
Starbase: SQLite on the Edge
https://starbasedb.com/
48 comments
I don't really understand what this is offering beyond Cloudflare's recent release of running SQLite in durable objects: https://blog.cloudflare.com/sqlite-in-durable-objects/. Is it about providing an external interface to Cloudflare's SQLite databases?
The project is open source (https://github.com/Brayden/starbasedb/blob/main/src/index.ts). Yes to it provides a way to update Cloudflare's SQLite with HTTP.
If that's the case there's libsql (https://github.com/tursodatabase/libsql) which already provides HTTP clients, embedded replicas (essentially good old SQLite but with replication support), self-hosted SQLite server and is maintained by a company using it for their own product.
Some day I really need to learn when to use sqlite in a durable object vs the eventually consistent one (r2).
iirc they are both powered by the same engine to stream and replicate the WAL. I believe R2 is now implemented as a Durable Object backed by SQLite now.
Does this mean that R2 is not "eventually consistent" anymore?
I wonder what are the use cases (and when it's safe) to use "eventually consistent".
I'm guessing that maybe things like social media posts could be fine with "eventually consistent". It's not really critical to have the latest data.
I'm guessing that things like a shopping cart, a user account, a session shouldn't use a "eventually consistent" database.
I wonder what are the use cases (and when it's safe) to use "eventually consistent".
I'm guessing that maybe things like social media posts could be fine with "eventually consistent". It's not really critical to have the latest data.
I'm guessing that things like a shopping cart, a user account, a session shouldn't use a "eventually consistent" database.
Still think there is a lot we can add to StarbaseDB to make the developer experience on SQLite databases better, and personally I think it starts with a better external interface. Provide a frictionless way to get started. Then figuring out how to improve developer experiences on how they interface with databases.
Is it auto-accessible REST endpoints? Easy to implement websocket support for the database? Data replication for scaling reads? Offline data syncing? A lot of potential wins for a layer like this to build on.
Is it auto-accessible REST endpoints? Easy to implement websocket support for the database? Data replication for scaling reads? Offline data syncing? A lot of potential wins for a layer like this to build on.
How does this compare with Cloudflare’s new offering SQLite it durable objects: https://blog.cloudflare.com/sqlite-in-durable-objects
So it actually sits on top of it, providing an HTTP interface to interact with them.
Not a bad idea. There'll always be someone who says "but I can already do that with X" and then someone who replies to that with "but I can already do that with Y" and in the end when you're done trying to stick to the purist laws you've ran out of time and motivation and your project goes into the Pile of Dead Projects.
Or, we have easy to use, enjoyable interfaces and let ourselves use them, and actually finish things.
Or, we have easy to use, enjoyable interfaces and let ourselves use them, and actually finish things.
From what I could understand of their architecture [1], you may lose data when the machine running DO fails.
DO implements a very comprehensive strategy to mitigate data loss in DO backed by SQLite. But Starbase seems to add a layer of query caching on the application level, which seems to be naively implemented (I might be misunderstanding, please correct if I'm wrong).
[1] https://starbasedb.com/blog/developing-acid-transaction-supp...
DO implements a very comprehensive strategy to mitigate data loss in DO backed by SQLite. But Starbase seems to add a layer of query caching on the application level, which seems to be naively implemented (I might be misunderstanding, please correct if I'm wrong).
[1] https://starbasedb.com/blog/developing-acid-transaction-supp...
We don't cache query responses in the Starbase application layer. When we originally set out to build this, we were under the assumption we had to handle the ACID transactional support ourselves. Thankfully someone from Cloudflare shed some light for us as you can see in our Github issues and open PR's :)
Learn more about that here: https://github.com/Brayden/starbasedb/issues/12
Learn more about that here: https://github.com/Brayden/starbasedb/issues/12
Just fyi, the person that replied wasn't just "someone" at Cloudflare. It was Kenton Varda (kentonv here). He's the creator of Cloudflare workers, he's an incredible engineer.
On the other hand, everyone is just someone, somewhere. Let's not criticise people for not giving enough of an intro to people.
Yes, of course! Sorry didn't mean to make it seem insignificant – was quite excited to see him show up there from a personal fanboy perspective.
This is pretty cool. Great artwork too. Building edge DBs on top of S3 or R2 had interested me for a long time. Thanks for writing it!
Update: bug in redirection: https://starbasedb.com/blog/blog/blog/blog/blog/blog/develop...
Update: bug in redirection: https://starbasedb.com/blog/blog/blog/blog/blog/blog/develop...
+1. Love the cassette falling in place with the sound effect. Still don't know what it actually does though, but that is very satisfying.
Had no idea this was put on HN today. Author of StarbaseDB here, will get a bug fix for the blog redirection pushed out soon! Thanks for bringing this up.
This is awesome. We’ve adopted a similar architecture with DO. Does it do data sync across devices? Offline first? Or is it all online thru DO and D1?
DO = Cloudflare Durable Objects, https://developers.cloudflare.com/durable-objects/
D1 = Cloudflare D1, https://developers.cloudflare.com/d1/
Just incase someone else was as lost as me.
D1 = Cloudflare D1, https://developers.cloudflare.com/d1/
Just incase someone else was as lost as me.
ah, DO always translates to Digital Ocean for me. So thanks for that! acronyms with name collisions are the woooorst
Cloudflare recently updated this:
Zero-latency SQLite storage in every Durable Object
https://blog.cloudflare.com/sqlite-in-durable-objects/
Zero-latency SQLite storage in every Durable Object
https://blog.cloudflare.com/sqlite-in-durable-objects/
What is the difference between this and Cloudflare’s first party D1 database offering? https://developers.cloudflare.com/d1/
D1 is SQLite
D1 is SQLite
It's actually sitting on top of Cloudflare's new SQLite storage in Durable Objects. You can learn more here:
https://blog.cloudflare.com/sqlite-in-durable-objects/
There's actually a section called: What about D1?
https://blog.cloudflare.com/sqlite-in-durable-objects/
There's actually a section called: What about D1?
Really glad Kenton chimed in there, as it wasn't clear in the docs and I was about to implement a similar pseudo transaction pattern as this.
Pretty excited about this tech, it really flips the script on the concept of a DB per user / sharding approach which used to be costly at the schema/physical db level.
Pretty excited about this tech, it really flips the script on the concept of a DB per user / sharding approach which used to be costly at the schema/physical db level.
He was mentioned, but did he comment here?
He commented here: https://github.com/Brayden/starbasedb/issues/12
Am I right to say that this is a template on top of cloudflare durable objects, not an actual software of its own?
This is very appealing. If the authors are reading:
What is the architecture overview? Does each worker have a copy or do all of them work on the same DB files? How does it handle concurrent workers and db locking? Is SQLite native locking mechanism sufficient in this application?
What is the architecture overview? Does each worker have a copy or do all of them work on the same DB files? How does it handle concurrent workers and db locking? Is SQLite native locking mechanism sufficient in this application?
Multiple workers would connect to the same durable object and write to it, basically, the Durable Object takes care of all the writes in a single thread, so there's no issues with concurrent modifications.
We probably need to build out some sort of data replication to scale reads though Durable Objects are limited to 1000 RPS.
We probably need to build out some sort of data replication to scale reads though Durable Objects are limited to 1000 RPS.
I find the website design to be quite charming. Actually evokes some of the Outer World game vibes.
I found it to be hilariously uninformative.
I have this old school expectation that I should be able to understand what your website/product/project is about but reading the first few sentences in the "above the fold" section of your homepage. I don't want to play a game or click below the fold "read more" links to try and find out what this web content has to do with the domain name "starbasedb".
Perhaps they actually know their intended (seemingly game developer?) audience that I give them credit for? But I do have plans (admittedly only very vague plans right now) to use one of the available "SQLite from javascript in a webpage querying the SQLite file in an object store like S3 without a web backend" techniques or projects. Maybe they'd work for me with my non game oriented use case? I don't know, it seems like they don't want me to find out...
I have this old school expectation that I should be able to understand what your website/product/project is about but reading the first few sentences in the "above the fold" section of your homepage. I don't want to play a game or click below the fold "read more" links to try and find out what this web content has to do with the domain name "starbasedb".
Perhaps they actually know their intended (seemingly game developer?) audience that I give them credit for? But I do have plans (admittedly only very vague plans right now) to use one of the available "SQLite from javascript in a webpage querying the SQLite file in an object store like S3 without a web backend" techniques or projects. Maybe they'd work for me with my non game oriented use case? I don't know, it seems like they don't want me to find out...
Appreciate the candidness to your response here. Honestly we put this brand and website together in two days and we wanted to have some fun with it. That said there is a lot more we can do to make the website more informative, and we'll be sure to improve it as we go. Expect more blog posts, tutorials and implementation guides this week.
ELI5: Does "Open source, scale-to-zero, HTTP SQLite database built on top of Cloudflare Durable Objects" mean that you should pay and/or use Cloudflare for this? Or CDOs are generally available outside Cloudflare?
So right now you will have to pay and use Cloudflare for this, there's no other way around it today.
Any comparison with Dzero?
https://news.ycombinator.com/item?id=40563729
https://news.ycombinator.com/item?id=40563729
Does stuff like this still have database-level locking? If so, how does that scale at all?
No, there's no traditional database-level locking here. The Durable Object itself is single-threaded, so you don't have to worry about multiple things trying to change the same data at the same time.
As far as scaling yeah there are still some challenges there. I mentioned in another response there's a limit of 1K RPS, and the single-threadedness could become a bottle-neck. I think there are probably clever workarounds here that we still want to explore, for instance each Durable Object could handle a seperate piece of state and you would effectively have a level of state-based sharding.
As far as scaling yeah there are still some challenges there. I mentioned in another response there's a limit of 1K RPS, and the single-threadedness could become a bottle-neck. I think there are probably clever workarounds here that we still want to explore, for instance each Durable Object could handle a seperate piece of state and you would effectively have a level of state-based sharding.
May I just say; hell yeah on the brand identity! - finally someone has the guts to actually have an identity instead og pastel based standard b2b shit.
Last time I saw that was zendesk with their Buddha's and weird tonal audio devices.
Edited for corrected semi-colon because I'm haunted by a grammar-nazi.
Last time I saw that was zendesk with their Buddha's and weird tonal audio devices.
Edited for corrected semi-colon because I'm haunted by a grammar-nazi.
I love the website design than the product.
`Isolation. Due to the nature of our queue and query executions, everything is synchronous by nature so no two queries can be ran concurrently making them run in an isolated nature by default. Check.`
I'm not sure if I misunderstood something, but using a global queue/lock to enforce isolation properties seems to be just a little disingenuous for a dbms. It works, but it's odd to talk about isolation in a system with zero concurrency. It's like saying a bicycle gets infinite miles per gallon in fuel efficiency.
I'm not sure if I misunderstood something, but using a global queue/lock to enforce isolation properties seems to be just a little disingenuous for a dbms. It works, but it's odd to talk about isolation in a system with zero concurrency. It's like saying a bicycle gets infinite miles per gallon in fuel efficiency.
This has the same name as an Elon Musk project. I can see the point of reusing names (this is a clever spin after all), but I’d expect the association to evoke quite some negative reactions.