SapphireDb – Open-Source Alternative to Firebase(sapphire-db.com)
sapphire-db.com
SapphireDb – Open-Source Alternative to Firebase
https://sapphire-db.com
62 comments
Is it supposed to be self-hosted? Wasn’t clear to me after taking a quick look.
Yes, it can be. I realise it's not too obvious so I'll add the documentation for self hosting over the next 1 week
Thanks! Will keep an eye on it.
That looks preeeetty cool. Keepin an eye on it.
Neat. Did you have a look at Hasura before you implemented this, or does your solution have other features you needed?
Edit: also, Debezium might be a good backend for this, that way you would be able to support other databases than Postgres.
Edit: also, Debezium might be a good backend for this, that way you would be able to support other databases than Postgres.
I took a lot of inspiration from Hasura and Debezium. At the time they used triggers to send events via NOTIFY (not sure if they still do)
The issue with NOTIFY is that it has an 8000 byte limit. Some of our updates are large (JSONB columns) so we need more than 8000 bytes, or we would have to notify just the row identifier, then fetch the data (inefficient).
Supabase hooks in to Postgres' replication stream so there is no issue. It converts the streaming bytes to JSON.
One other bonus - you don't have to set a trigger on every table. Just run `CREATE PUBLICATION supabase_realtime ON ALL TABLES`, then you're good to go.
The issue with NOTIFY is that it has an 8000 byte limit. Some of our updates are large (JSONB columns) so we need more than 8000 bytes, or we would have to notify just the row identifier, then fetch the data (inefficient).
Supabase hooks in to Postgres' replication stream so there is no issue. It converts the streaming bytes to JSON.
One other bonus - you don't have to set a trigger on every table. Just run `CREATE PUBLICATION supabase_realtime ON ALL TABLES`, then you're good to go.
Debezium uses neither triggers nor NOTIFY. For Postgres, it uses one of several supported logical decoding plug-ins (pgoutput, decoderbufs, wal2json), which doesn't have any size limit and also allow to capture changes occuring while it isn't running by catching up with changes from the WAL from the point where it left off before.
Disclaimer: Debezium lead here
Disclaimer: Debezium lead here
That makes a lot of sense. Tempted to try this out now. If people would just stop coming up with all those really interesting projects, that'd be great. At some point I should probably get some actual work done...
Creator of Redwood here, which is similar in many ways to this and GunDB (hi Mark!). Agreed that we need more projects like this.
Redwood might be better described as a "p2p application platform" than a realtime database, but there's plenty of overlap. It's written in Go, supports HTTPS, Libp2p, and WebRTC transports (and soon, a Cap'n Proto RPC interface). It can also:
- serve as a Git remote
- stand in as an application server (you can upload assets directly to it, and they're served over regular HTTPS)
- execute Lambda-like functions written in Go, Lua, and Javascript that can be installed at any point in a given state tree
Would love some early feedback, and also to discuss similarities and differences with Sapphire's model. I've set up some demos in the "scenarios" folder -- a chat app, a realtime document editor, and a Git integration. Take a look if you're interested:
https://github.com/brynbellomy/redwood
Redwood might be better described as a "p2p application platform" than a realtime database, but there's plenty of overlap. It's written in Go, supports HTTPS, Libp2p, and WebRTC transports (and soon, a Cap'n Proto RPC interface). It can also:
- serve as a Git remote
- stand in as an application server (you can upload assets directly to it, and they're served over regular HTTPS)
- execute Lambda-like functions written in Go, Lua, and Javascript that can be installed at any point in a given state tree
Would love some early feedback, and also to discuss similarities and differences with Sapphire's model. I've set up some demos in the "scenarios" folder -- a chat app, a realtime document editor, and a Git integration. Take a look if you're interested:
https://github.com/brynbellomy/redwood
> and soon, a Cap'n Proto RPC interface
This is interesting, and good to hear. Are there still a lot of projects adding capnproto support? It's a great protocol but I've worried it might be dying.
This is interesting, and good to hear. Are there still a lot of projects adding capnproto support? It's a great protocol but I've worried it might be dying.
Dying? Hm, not to my knowledge. In fact, I saw a highly-commented thread on HN just a couple of weeks ago about Cap'n Proto. What gives you the sense that it's on its way out?
At the least, it may be reasonable to also consider a gRPC iface (if you don't already have it).
Sandstorm Oasis going away and the last time I checked (maybe a year ago?) work seemed to have stopped on getting the protocol implemented in the browser.
Hi, I'm the author of Cap'n Proto, and co-founder of Sandstorm.io. When Sandstorm failed, I went to Cloudflare where I'm leading the Cloudflare Workers project. We're using Cap'n Proto (including RPC) pretty heavily there!
That said, it's true that Cap'n Proto relies on its users to contribute improvements to the implementation. We don't currently have a team who can dedicate their time to building out capnp implementations in every language.
At present, within Cloudflare, the heaviest use is C++, with some stuff in Go, Rust, Java, and Lua. Hence, if you look at the C++ implementation you'll see it's been pretty active. At the moment we don't have a JavaScript use case so haven't put effort into that implementation. But given that Workers itself is a JavaScript platform, it seems reasonably likely that we'll end up putting some work into the JavaScript/TypeScript implementation at some point down the road.
That said, it's true that Cap'n Proto relies on its users to contribute improvements to the implementation. We don't currently have a team who can dedicate their time to building out capnp implementations in every language.
At present, within Cloudflare, the heaviest use is C++, with some stuff in Go, Rust, Java, and Lua. Hence, if you look at the C++ implementation you'll see it's been pretty active. At the moment we don't have a JavaScript use case so haven't put effort into that implementation. But given that Workers itself is a JavaScript platform, it seems reasonably likely that we'll end up putting some work into the JavaScript/TypeScript implementation at some point down the road.
Glad to hear it! It really would be a shame for such a great protocol to go unused.
That's interesting that you feel Sandstorm failed. I draw inspiration from it on a regular basis for several of my projects. Maybe it was just ahead of its time.
That's interesting that you feel Sandstorm failed. I draw inspiration from it on a regular basis for several of my projects. Maybe it was just ahead of its time.
Thanks for the kind words.
Sandstorm the company definitely failed (ran out of money and couldn't raise more, while having negligible actual revenue). Sandstorm the open source project continues and has actually seen an uptick in development activity lately.
Sandstorm the company definitely failed (ran out of money and couldn't raise more, while having negligible actual revenue). Sandstorm the open source project continues and has actually seen an uptick in development activity lately.
Just wanted to thank you for your brilliant work!
> Would love some early feedback
Nice project! I had considered building something similar (actually just services discoverable via DHT, nothing concerning storage). Can you discuss here or in the README its resiliency toward bad actors? Specifically, how does it work in a p2p environment with untrusted clients? If it's still an unknown, just a note on whether it's even a threat model you're developing within would be helpful.
Nice project! I had considered building something similar (actually just services discoverable via DHT, nothing concerning storage). Can you discuss here or in the README its resiliency toward bad actors? Specifically, how does it work in a p2p environment with untrusted clients? If it's still an unknown, just a note on whether it's even a threat model you're developing within would be helpful.
Sure! Nailing down a robust security model is a major goal for Redwood. There are currently a few different mechanisms, some already partially built, and others still in the research stage.
For what it's worth, I have a silly side project goal of reimplementing Bitcoin on top of Redwood, just to prove it can provide the same (general) security guarantees. Not sure I'll get around to that any time soon, though :)
Anyway, here's what we do:
1. Users are identified by a public/private keypair (actually using Ethereum's implementation at the moment, just so I could focus on other stuff). All transactions must be signed by the sender, allowing recipients to verify the sender identities.
2. You can place "transaction validators" at any node in your state trees, and any transaction affecting the subtree under the validator will be checked by that validator. Currently, there's a "permissions" validator included that gives a simple way to control writes based on the Ethereum keypair I mentioned above. The current implementation of the permissions validator isn't very well fleshed out yet -- it just lets you specify a true/false write permission associated with a [user, keypath regex] pair -- but it's a decent placeholder for now, until I can get around to roles and policies.
3. You can also write custom transaction validators in Go/Lua/Javascript, which should make it trivial to implement just about any access control model you desire.
4. You can also create a "private" tree by explicitly specifying the set of users who are allowed to read from and write to that tree. The default Redwood node implementation does automatic peer discovery and keeps a list of peers whose identities/addresses have been verified. When it receives a transaction for a private tree, it only gossips that transaction to the tree's members (as opposed to its behavior with public trees, which is to gossip transactions to any peer who subscribes to that tree).
5. We also want secure persistent storage for transactions so that we can get high availability and redundancy without compromising the security model. To facilitate this, we allow you to configure the node to talk to what we're calling a "remote blind store" (essentially a key-value DB running on a separate piece of hardware, possibly off-site). The Redwood node encrypts transactions with a key the remote store doesn't possess and sends it over gRPC to the blind store.
6. Some applications will have a high volume of transaction data, and will want to be able to prune/merge/rebase that data to save space. To prevent bad actors from issuing malicious prune requests, the remote blind stores are going to implement their own p2p protocol involving threshold signatures. Prune requests will only be honored if a quorum of these blind stores sign off on them.
We've been thinking about this quite a bit, but not publicly. I'd love for someone to help us punch holes in it!
For what it's worth, I have a silly side project goal of reimplementing Bitcoin on top of Redwood, just to prove it can provide the same (general) security guarantees. Not sure I'll get around to that any time soon, though :)
Anyway, here's what we do:
1. Users are identified by a public/private keypair (actually using Ethereum's implementation at the moment, just so I could focus on other stuff). All transactions must be signed by the sender, allowing recipients to verify the sender identities.
2. You can place "transaction validators" at any node in your state trees, and any transaction affecting the subtree under the validator will be checked by that validator. Currently, there's a "permissions" validator included that gives a simple way to control writes based on the Ethereum keypair I mentioned above. The current implementation of the permissions validator isn't very well fleshed out yet -- it just lets you specify a true/false write permission associated with a [user, keypath regex] pair -- but it's a decent placeholder for now, until I can get around to roles and policies.
3. You can also write custom transaction validators in Go/Lua/Javascript, which should make it trivial to implement just about any access control model you desire.
4. You can also create a "private" tree by explicitly specifying the set of users who are allowed to read from and write to that tree. The default Redwood node implementation does automatic peer discovery and keeps a list of peers whose identities/addresses have been verified. When it receives a transaction for a private tree, it only gossips that transaction to the tree's members (as opposed to its behavior with public trees, which is to gossip transactions to any peer who subscribes to that tree).
5. We also want secure persistent storage for transactions so that we can get high availability and redundancy without compromising the security model. To facilitate this, we allow you to configure the node to talk to what we're calling a "remote blind store" (essentially a key-value DB running on a separate piece of hardware, possibly off-site). The Redwood node encrypts transactions with a key the remote store doesn't possess and sends it over gRPC to the blind store.
6. Some applications will have a high volume of transaction data, and will want to be able to prune/merge/rebase that data to save space. To prevent bad actors from issuing malicious prune requests, the remote blind stores are going to implement their own p2p protocol involving threshold signatures. Prune requests will only be honored if a quorum of these blind stores sign off on them.
We've been thinking about this quite a bit, but not publicly. I'd love for someone to help us punch holes in it!
Nice work! :)
If you are looking for an open-source replacement of the Firebase Realtime Database, you might also be interested in Icepeak (https://github.com/channable/icepeak) (full disclosure, I'm one of the authors).
Icepeak was written because we were unsatisfied with the reliability and with the (occasionally) high latency of Firebase, both for pushing updates via HTTP (>1s for some requests) and for receiving updates over websockets.
This was about 3 years ago though, so I hope that Google improved the quality of the service in the mean time.
Icepeak was written because we were unsatisfied with the reliability and with the (occasionally) high latency of Firebase, both for pushing updates via HTTP (>1s for some requests) and for receiving updates over websockets.
This was about 3 years ago though, so I hope that Google improved the quality of the service in the mean time.
Firebase and AppEngine Datastore have both been superceded by Firestore, which is has the realtime features of Firebase, but the scalability and performance (better, I think) of Datastore, and more data modeling and query flexibility.
That sounds interesting. Have you compared your service to pouchdb and gun? Is it possible to use icepeak in offline mode and automatically sync when getting online again?
[1] https://pouchdb.com/
[2] https://gun.js.org/
[1] https://pouchdb.com/
[2] https://gun.js.org/
PouchDB looks like the client side library.
Do you still need CouchDB on the backend?
I see from the mailing list it is still a very active project, which surprised me.
Do you still need CouchDB on the backend?
I see from the mailing list it is still a very active project, which surprised me.
If you want to have PouchDB without being tied to the CouchDB-backend, check out RxDB. It supports replication via GraphQL.
https://github.com/pubkey/rxdb
https://github.com/pubkey/rxdb
PouchDB is CouchDB-Protocol implemented in javascript so you can run it wherever there is javascript, not just as a client. If you want to have a backend, you can use any DB that supports that protocol (there are several), which includes using a PouchDB instance run by node.js.
Is PouchDB + CouchDB considered an alternative to Firebase?
That depends on your use case since Firebase is more than a database. Since CouchDB is from 2005 and Firebase is from 2011, it is in comparison a very mature product and protocol worth trying.
Hi! What are you using for conflict resolution? Is it CRDT? Have you seen RON? https://github.com/ff-notes/ron
In the Stackblitz example the Angular client is connecting directly to the SapphireDB database with credentials stored in the app.module.ts.
Could you build a scalable real world, production application this way?
With no in the middle NodeJS web server handing business logic? Handle all the business logic in the Angular client and just CRUD the objects back to SapphireDB? Essentially just Client--> SapphireDB
Or am I missing something here? I see there's a asp.net configuration, which would be Client--> Server--> SapphireDB
Is there a package or support for connecting a NodeJS server to SapphireDB database?
Also, is this production ready?
Could you build a scalable real world, production application this way?
With no in the middle NodeJS web server handing business logic? Handle all the business logic in the Angular client and just CRUD the objects back to SapphireDB? Essentially just Client--> SapphireDB
Or am I missing something here? I see there's a asp.net configuration, which would be Client--> Server--> SapphireDB
Is there a package or support for connecting a NodeJS server to SapphireDB database?
Also, is this production ready?
The credentials you found in the app.module.ts are just credentials to identify the client (not the user). It is just to give an extra layer of security and only allow specific applications access to the server. User authentication is handled through tokens for example.
The architecture looks more like this: Client -> Asp.Net Core server with SapphireDb -> any database server
SapphireDb is not a database. Its more like the connector between database and clients. Because of that there is no package for a NodeJS because it would just be proxy.
Yes I think it is production ready. I'll add a section in the docs describing the criteria this opinion is based on.
The architecture looks more like this: Client -> Asp.Net Core server with SapphireDb -> any database server
SapphireDb is not a database. Its more like the connector between database and clients. Because of that there is no package for a NodeJS because it would just be proxy.
Yes I think it is production ready. I'll add a section in the docs describing the criteria this opinion is based on.
Firebase is a collection of services. This seems to be an alternative to the Firebase Realtime Database specifically
This is not even true. Firebase Realtime Database supports offline-first while SapphireDB is just streaming results to the client, like RethinkDB or Meteor.
Offline-first is already a planned feature and only missing because of the lack of development capacity.
Which is sort of a legacy database they keep probably more for compatibility reasons. Firestore is the actual database used by most new apps.
Another open source "realtime" database, which supports joins, has a nice admin interface, sharding, and realtime queries:
https://rethinkdb.com
It's what mongo should have been from the beginning -- they won the correctness battlew hile mongo won the marketing war.
I should mention that I've had some colleagues run into problems with scaling it so it's not all roses but a database that scales painlessly in every case is AFAIK a fantasy.
https://rethinkdb.com
It's what mongo should have been from the beginning -- they won the correctness battlew hile mongo won the marketing war.
I should mention that I've had some colleagues run into problems with scaling it so it's not all roses but a database that scales painlessly in every case is AFAIK a fantasy.
Need to followup, but angular-only examples... would be nice to see a straight/simple client that can be used with, for example, angular listed...
It is probably out there already.
It is probably out there already.
Was building an opensource version cheaper for you than just using firebase? I'm generally curious. I commonly see OS alternatives to paid services but I've never seen why it was made. Is it cheaper to run and maintain your own servers? Are longer outages and downtimes ok for your company? Security vulnerabilities? Maybe these are things you don't care about and that's fine.
It would be nice to see "Use this if.." section.
It would be nice to see "Use this if.." section.
based on .net core, for what it's worth
Which is a bit weird since .NET Core has standardized on the open source SignalR, which is really good. It would make more sense to build on top of that.
> SapphireDb should serve as a self hosted alternative to firebase and also gives you an alternative to SignalR.
I took SignlaR into consideration when building that.
SignalR does not have some client implementations. I decided to create a small protocol that is easily adoptable in many technologies.
SignalR also has some limitations that made it impracticable for the project.
SignalR does not have some client implementations. I decided to create a small protocol that is easily adoptable in many technologies.
SignalR also has some limitations that made it impracticable for the project.
depends on your audience
Needs a base npm library, that is not tethered to Angular.
https://github.com/SapphireDb/SapphireDb/issues/3
https://github.com/SapphireDb/SapphireDb/issues/3
This is planned and will come soon.
Cool - I'm glad to see more tech similar to Firebase pop up that's self hosted.
The big request would be mobile. I mostly use FB for the web but knowing that I could create a mobile version using the same platform would be great.
The big request would be mobile. I mostly use FB for the web but knowing that I could create a mobile version using the same platform would be great.
"SapphireDb also supports running in an NLB with multiple instances and scales very good."
In English, when speaking of actions you must say "well" instead of "good", ie. "scales very well". While there are a million weird things about English, this particular mistake is used as a cultural marker of "childish" or "low-intelligence" speech, so you probably want to fix it ASAP.
Also what is an NLB???
In English, when speaking of actions you must say "well" instead of "good", ie. "scales very well". While there are a million weird things about English, this particular mistake is used as a cultural marker of "childish" or "low-intelligence" speech, so you probably want to fix it ASAP.
Also what is an NLB???
I'm a competitor, and I want to applaud this work!
- SapphireDB is truly Open Source, MIT!
- There demo worked like a charm!
We could really use more projects like this out there.
Here is a criticism though:
- It seems currently to only support Angular (?), which is gonna harm its adoption.
My recommendation would be to commit to Svelte, it seems like it will be the "next big thing" in the JS world, and could help drive adoption.
Obviously, this is something we're trying to do as well (https://gun.eco/docs/Svelte). If you're looking for a Firebase alternative, GUN is used in production with 11M+ downloads a month, but is NodeJS based so it won't work for the .NET audience that Sapphire is targeting.
What I think would be cool to add: (from my bias)
- Support graph data.
- Work offline / local-first.
Either way, thanks for building & posting about Sapphire, it is nice to see there is still big demand for Open Source Firebase alternatives!
- SapphireDB is truly Open Source, MIT!
- There demo worked like a charm!
We could really use more projects like this out there.
Here is a criticism though:
- It seems currently to only support Angular (?), which is gonna harm its adoption.
My recommendation would be to commit to Svelte, it seems like it will be the "next big thing" in the JS world, and could help drive adoption.
Obviously, this is something we're trying to do as well (https://gun.eco/docs/Svelte). If you're looking for a Firebase alternative, GUN is used in production with 11M+ downloads a month, but is NodeJS based so it won't work for the .NET audience that Sapphire is targeting.
What I think would be cool to add: (from my bias)
- Support graph data.
- Work offline / local-first.
Either way, thanks for building & posting about Sapphire, it is nice to see there is still big demand for Open Source Firebase alternatives!
Offtopic:
> to commit to Svelte, it seems like it will be the "next big thing" in the JS world
How long does it usually take? In the JS world? Serious question as I am not in the JS world but reading HN/Reddit I would believe that things change quite fast. Someone told me over 1 year ago Svelte would be the next big thing and I see ‘no-one’ using it besides some hardcore enthusiasts; it is even hardly ever mentioned here or on Reddit; if it was a the next big thing, I think it would be?
Sorry for being uninformed, I am just curious.
> to commit to Svelte, it seems like it will be the "next big thing" in the JS world
How long does it usually take? In the JS world? Serious question as I am not in the JS world but reading HN/Reddit I would believe that things change quite fast. Someone told me over 1 year ago Svelte would be the next big thing and I see ‘no-one’ using it besides some hardcore enthusiasts; it is even hardly ever mentioned here or on Reddit; if it was a the next big thing, I think it would be?
Sorry for being uninformed, I am just curious.
the next best thing is hard to predict, as thre sister comment says. it's also not an important measure. the measure that matters is wheter a framework is going to stick around and be supported for the next few years.
to choose a framework look at all that are being used and recommended.
for every recommended framework there are a dozen that remain in obscurity.
that should narrow it down to a dozen or so. look at each and pick on you like the most by whatever metric you prefer. it does't matter.
keep using that same framework as long as you are happy with it.
if you are able to stick with the same framework for a few years you made good choice.
as for svelte, one metric is the stateofjs survey, which lists svelte as one of the top 6 frameworks.
it was added to the survey last year which shows a significant amount of interest. let's see if that is still true next year.
https://2019.stateofjs.com/front-end-frameworks/
my own choice, aurelia, which i picked a few years ago, is the 3rd most write-in choice. it has a healthy community and active development. and i see no need to change for the forseeable future. it doesn't need to be the next big thing.
to choose a framework look at all that are being used and recommended.
for every recommended framework there are a dozen that remain in obscurity.
that should narrow it down to a dozen or so. look at each and pick on you like the most by whatever metric you prefer. it does't matter.
keep using that same framework as long as you are happy with it.
if you are able to stick with the same framework for a few years you made good choice.
as for svelte, one metric is the stateofjs survey, which lists svelte as one of the top 6 frameworks.
it was added to the survey last year which shows a significant amount of interest. let's see if that is still true next year.
https://2019.stateofjs.com/front-end-frameworks/
my own choice, aurelia, which i picked a few years ago, is the 3rd most write-in choice. it has a healthy community and active development. and i see no need to change for the forseeable future. it doesn't need to be the next big thing.
I don't think this is clear cut. React has been around for more than 6 years. According to the latest State of JS survey[1], 72% of respondents have used React before and would use it again. That number is 7% for Svelte.
Svelte has a lot of interest, but it also has a very long way to go against a popular, 6 year old project.
[1]: https://2019.stateofjs.com/front-end-frameworks/
Svelte has a lot of interest, but it also has a very long way to go against a popular, 6 year old project.
[1]: https://2019.stateofjs.com/front-end-frameworks/
React was like this for a long while too, compared to Angular.
Then once some big "used in production by NYT/x/y/z" hits, a premier conference talk, then things explode with everyone virtue-signaling how they use the modern/best stack, floods of article/blogs are posted about how they switched over before you did, and the FOMO cargo cult bandwagon rocketships.
Svelte does look legit, but it is better to not buy into the JS virtue culture.
Then once some big "used in production by NYT/x/y/z" hits, a premier conference talk, then things explode with everyone virtue-signaling how they use the modern/best stack, floods of article/blogs are posted about how they switched over before you did, and the FOMO cargo cult bandwagon rocketships.
Svelte does look legit, but it is better to not buy into the JS virtue culture.
Thank you for your feedback.
An implementation for Svelte is already planned, but thank you for the recommendation. I'll move the codebase of the Angular implementation into a seperate project soon which will make integrations for other JS frameworks pretty easy.
Your project looks very interesting and I'll definitely take a look into it.
The support for graph data is still in consideration. Offline/Local first support is planned and will definitely be a feature in the future.
Big thanks for the recommendations.
An implementation for Svelte is already planned, but thank you for the recommendation. I'll move the codebase of the Angular implementation into a seperate project soon which will make integrations for other JS frameworks pretty easy.
Your project looks very interesting and I'll definitely take a look into it.
The support for graph data is still in consideration. Offline/Local first support is planned and will definitely be a feature in the future.
Big thanks for the recommendations.
Awesome, glad to hear! Keep up the good work!
Hey Mark. Sorry to go OT and I promise I'm usually not that guy, but the GUN site is really distracting with the flashing colors.
EDIT: This one, which I grabbed from your profile => https://gun.eco/
EDIT: This one, which I grabbed from your profile => https://gun.eco/
[deleted]
The hardest part was the real-time functionality. I ended up writing an elixir server that listens to Postgres’ built-in replication stream, converts it into JSON, then sends it over websockets.