Fluree DB – A scalable blockchain database(flur.ee)
flur.ee
Fluree DB – A scalable blockchain database
https://flur.ee
73 comments
Isn't blockchain just a slow database in the first place. What do you mean by blockchain meet database.
Having a multi-node database that syncs (even slowly) and handles merge conflicts is useful in many applications.
Specifically, this is useful in global health situations where you have lots of data collectors with tablets entering data that all has to eventually come together. But has lots of offline/online changes and has an untrusted data entry issue where you don’t want anything deleted.
Specifically, this is useful in global health situations where you have lots of data collectors with tablets entering data that all has to eventually come together. But has lots of offline/online changes and has an untrusted data entry issue where you don’t want anything deleted.
We have lots of distributed databases with merging/syncing abilities. You can also append-only and disable delets. No blockchain necessary.
I don't get your use-case regarding data collection people, what exactly will a blockchain prevent them from doing? Are they all entering the same piece of data to then pick the version they all input the most?
I don't get your use-case regarding data collection people, what exactly will a blockchain prevent them from doing? Are they all entering the same piece of data to then pick the version they all input the most?
It’s very likely that I’m just not aware. What databases enforce no deletes or edits in a verifiable way? I would appreciate you sharing some examples.
In my use case you have say 5 people covering different territory collecting info that usually doesn’t conflict. Imagine doing a census of a few villages. You don’t want data changed or altered especially time stamps and want it to be shared across all nodes when they are in proximity so they can sync to a remote instance when back in coverage. In some situations it’s literally the group meets at a tree at noon with a coordinator that syncs up all the collectors and moves on to the next group.
There are some conflicts that need to be merged, but mostly just syncing lots of nodes asynchronously and you don’t want anyone dropping records, maliciously editing them, or accidentally overwriting.
In my use case you have say 5 people covering different territory collecting info that usually doesn’t conflict. Imagine doing a census of a few villages. You don’t want data changed or altered especially time stamps and want it to be shared across all nodes when they are in proximity so they can sync to a remote instance when back in coverage. In some situations it’s literally the group meets at a tree at noon with a coordinator that syncs up all the collectors and moves on to the next group.
There are some conflicts that need to be merged, but mostly just syncing lots of nodes asynchronously and you don’t want anyone dropping records, maliciously editing them, or accidentally overwriting.
No deletes or edits is a really bad idea, especially for health data. In theory, it's pretty neat; in practice, it's a legal, compliance, and privacy nightmare.
How can it handle merge conflicts automatically?
I don’t need it to merge automatically, but storing them in a way that can be reconciled later is necessary. In some situations merges could be automatic if the records don’t have any conflicts.
You've described merging/replication which has nothing to do with blockchain.
That’s right, the blockchain part just adds in immutability of data and being able to trust different nodes. What database models have replication/merging models provide assurance of data integrity and perpetual versioning?
Blockchain adds neither things you mention. Both can be accomplished with digital signatures and timestamping, which are not new.
Blockchain alone doesn't prevent you from modifying data -- just rewrite the entire chain. As an example, Bitcoin only prevents this with proof of work, but only following accumulated difficulty prevents someone from thinking my privately-mined chain is true.
Blockchain alone doesn't prevent you from modifying data -- just rewrite the entire chain. As an example, Bitcoin only prevents this with proof of work, but only following accumulated difficulty prevents someone from thinking my privately-mined chain is true.
Not having read the white paper, it appears the defining difference here is data versioning.
I would assume it means that it's a database where it's impossible to modify the version history and maintain database integrity. It would offer some protection against tampering.
Normally I like this flowchart, but the permanent record is useful even within trusted parties. There aren’t many (any?) distributed databases with permanence built-in. So getting that for free is nice to not have to build yourself.
Especially self-hosted. If this allowed self hosted, it would be cool. Paying them seems not very useful.
Especially self-hosted. If this allowed self hosted, it would be cool. Paying them seems not very useful.
OMG, looks like „permanence“ is now to blockchainers what „web scale“ was to MongoDB fanboys.
Now if you were so kind to explain to me what about modern, distributed databases isn‘t „permanent“?
Permanence is the very definition of a database‘s job. If you really want to make extra sure it‘s permanent, give write-only permissions to the DB user, make backups and if you want to allow mutations, enable change data capture.
Now if there only was a way to short blockchain technology as a financial asset...
Now if you were so kind to explain to me what about modern, distributed databases isn‘t „permanent“?
Permanence is the very definition of a database‘s job. If you really want to make extra sure it‘s permanent, give write-only permissions to the DB user, make backups and if you want to allow mutations, enable change data capture.
Now if there only was a way to short blockchain technology as a financial asset...
What I mean by permanence is that you can’t delete. Databases typically allow delete and mod and this is a big problem for me.
What is a more accurate term? I always used persist for writing to disk, like databases do.
With the “workaround” you describe, although a big pain in the ass for lots of nodes, still doesn’t work. How do you know that someone’s backup is correct? Or how do you verify that a remote node didn’t write/delete/sync? There’s many situations that make your idea fail that a blockchain would work.
Although, I agree that lots of “blockchain” solutions just need a decent distributed cache since you don’t care much about integrity.
What is a more accurate term? I always used persist for writing to disk, like databases do.
With the “workaround” you describe, although a big pain in the ass for lots of nodes, still doesn’t work. How do you know that someone’s backup is correct? Or how do you verify that a remote node didn’t write/delete/sync? There’s many situations that make your idea fail that a blockchain would work.
Although, I agree that lots of “blockchain” solutions just need a decent distributed cache since you don’t care much about integrity.
Of course you can delete, just create a new DB and insert the records except the ones you don't want.
If you want to have integrity control, you can use digital signatures without blockchain.
If all you're trying to fix is making it hard to issue a DELETE statement, then make it easier to set permissions. Which isn't really hard in any mainstream database...
It seems like this is a log, which might work better, although existing RDBMS already have high performance versioning/change tracking.
If you want to have integrity control, you can use digital signatures without blockchain.
If all you're trying to fix is making it hard to issue a DELETE statement, then make it easier to set permissions. Which isn't really hard in any mainstream database...
It seems like this is a log, which might work better, although existing RDBMS already have high performance versioning/change tracking.
You can only delete up to the last point you synchronized. So once your record gets onto any other node, then it becomes harder and harder with more nodes to delete or change. Regardless of root/permissions/whatever.
Of course you can use signatures as you basically start recreating blockchain at that point (eg, each transaction is signed and linked to the previous; then connected with other nodes doing the same thing. A change to any segment would be detectable, etc etc).
Basically, it is just a distributed log that removes any change for changing data and the inability to set any permissions to remove data.
Maybe I’m being dense, but please let me know of any mainstream databases that give this functionality without relying on permissions to insure integrity. If an admin is able to change data, then that won’t work in my use case.
Of course you can use signatures as you basically start recreating blockchain at that point (eg, each transaction is signed and linked to the previous; then connected with other nodes doing the same thing. A change to any segment would be detectable, etc etc).
Basically, it is just a distributed log that removes any change for changing data and the inability to set any permissions to remove data.
Maybe I’m being dense, but please let me know of any mainstream databases that give this functionality without relying on permissions to insure integrity. If an admin is able to change data, then that won’t work in my use case.
Define "synchronized". If you already have a way for nodes to determine something was valid as of a certain set of data and time, you've already solved it.
I guess it means no undetectable delete. Like Git.
You can copy it, filter-branch it. But if something is deleted the hash will be different.
You can copy it, filter-branch it. But if something is deleted the hash will be different.
> There aren’t many (any?) distributed databases with permanence built-in.
Git.
Git.
Since one of these buzzword bingo projects are posted so often, can someone explain what this does, and why I shouldn't just use datomic anyways?
It does absolutely nothing interesting. It's a broken database with a buzzword attached
What do these projects hope to gain from pushing bad tech or broken ideas? Why do so many founders WANT to bust a Theranos-esque move? Its nuts
Money. Bad tech and broken ideas have never stopped businesses before and some have become rather successful. Using blockchain hype along with access to unregulated ICO markets just makes it easier.
Epicenter podcast with the founders for those interested: https://epicenter.tv/episode/216/
Seems more like a blockchain as a service like Chain's Sequence than an actual database.
IIIUC the fundemental datamodel here is a log of changes. A "query engine" is layered on top to get reasonable query performance, but that amounts to maintaining a materialized view for any data that might want to be queried. Otherwise the blockchain itself is not practically queriable by itself.
Sounds Datomic, but way less efficient.
Is my math right that a 1 gb database sitting there doing nothing is 1.4M tokens per month (2/MB/hour). And tokens are $5/200k, so $35/month before you do anything.
This seems pretty crazy. But I really need a blockchain db for the permanence.
This seems pretty crazy. But I really need a blockchain db for the permanence.
> But I really need a blockchain db for the permanence.
Do you really? What exactly is so permanent and who holds all the copies?
Do you really? What exactly is so permanent and who holds all the copies?
I explain it on the thread, but I have health records for disease outbreaks where the version is very important never to delete. The copies end up being stored amongst a few countries and Health partners. In some situations, countries can’t be trusted to archive or not alter records as it could be politically sensitive (eg, “maybe we just edit the db to say 10 cases of disease X is really 10 cases of disease Y.”). Or prevent situations where one site accidentally deletes some records, but not all, and it isn’t caught for weeks later where there have been other adds and edits over the time. Figuring out the restore was really hard. Being able to immediately detect changes across all nodes is valuable.
maybe you'd be interested in opentimestamps or factom, they use the bitcoin blockchain to timestamp documents
https://petertodd.org/2016/opentimestamps-announcement
https://petertodd.org/2016/opentimestamps-announcement
Me too. Have you find anything good so far?
I just learned of the gun,https://github.com/amark/gun project from a downthread post. I was kind of surprised that it didn’t exist and had some really janky sql+git stuff.
Somewhat related Ask HN-commenters:
Are there any widely used append-only DBMS's out there? Perhaps even with built in support for hash chains? I would like to have basically a single node, non-POW, "blockchain" database. Or, described differently, I guess, ≈≈ git for a database.
https://github.com/attic-labs/noms (full disclosure: my project)
(not widely used though)
git as a database is Datomic http://www.dustingetz.com/:datomic-in-four-snippets
You basically want a linked list.
Well, not really. It would be nice to be able to query the latest state in an efficient way, model relations and data constraints and have transaction handling etc as we are used to with traditional databases.
Ah, so you want CQRS + Event Sourcing?
i find it strange that none of the team members is an engineer
Brian Platz, Fluree CEO, is an entrepreneur and engineer. I know Fluree's written mostly in Clojure, and he wrote the core. (He worked out of the same co-working space I'm in but recently the company graduated to their own space.)
Brian Platz (https://angel.co/brian-platz) also previously founded SilkRoad Technology (not the other Silk Road), an HR management solution (among perhaps other things) ... https://www.silkroad.com/. Many institutions use Silk Road, e.g., the institution where my wife teaches, Wake Forest U ... here's their Silk Road portal: https://wakejobs.silkroad.com/WFU/Employment_Listings.html.
Brian Platz (https://angel.co/brian-platz) also previously founded SilkRoad Technology (not the other Silk Road), an HR management solution (among perhaps other things) ... https://www.silkroad.com/. Many institutions use Silk Road, e.g., the institution where my wife teaches, Wake Forest U ... here's their Silk Road portal: https://wakejobs.silkroad.com/WFU/Employment_Listings.html.
Immutable eh? What about GDPR?
I've not read too much into Fluree itself, but GDPR doesn't actually say you have to actually delete data. You can cryptographically shred it, among a few other options.
How do you cryptographically shred it in a immutable blockhain? And what are the other options?
It explicitly says you have to “erase the personal data”, GDPR Chapt 3 Art. 17
I think you would put the surrogate key in the block chain interactions, and then have a side place where you store the encrypted gdpr data. Then you can just overwrite the fields with null and keep the key.
So this is still a gray area, the law explicitly says that you should erase all data. So there can’t be any information that could that could be traced to you. Even if you do encrypt the data to let’s say “xuiidha” this is an identifier and therefore under the law.
If I look at the encrypted data before I delete the surrogate key and can see that this data is related to a person x. That’s an identifier. Even if I do delete that afterwards I can still connect that information because I’ve seen the data before. And even if you delete something there might be enough information left to still figure out who that person was and then under the law.
The right to be erased does mean that there shouldn’t be any traces of you ever been there in the first place and the law is pretty clear about it. It’s not the weaker term “deleted”. But I guess there has to a ruling in some court before we know how they interprete this law.
If I look at the encrypted data before I delete the surrogate key and can see that this data is related to a person x. That’s an identifier. Even if I do delete that afterwards I can still connect that information because I’ve seen the data before. And even if you delete something there might be enough information left to still figure out who that person was and then under the law.
The right to be erased does mean that there shouldn’t be any traces of you ever been there in the first place and the law is pretty clear about it. It’s not the weaker term “deleted”. But I guess there has to a ruling in some court before we know how they interprete this law.
If you can overwrite the data fields, then it is no longer immutable, is it?
There us no such thing as immutable data. The only constant is change. I'm suggesting that in the modeling of the data structures in a system, you rely only on the surrogate value and not the actual values. You're holding an immutable key to mutable data, that can be altered, but isn't necessary to be constant for the system to work. There is really no alternative, because laws are making it increasingly likely that immutable data stores will not be legally sound. Data stores like datomic have the notion of excision to handle this. Blockchains may require immutability, but that simply means you shouldn't put data in that must be removed by a court order.
This sounds a lot like (full disclosure: mine) https://github.com/amark/gun ? How do you compare?
- Except ours is Open Source (MIT/ZLIB/Apache2) and ranked #2 on GitHub in Blockchain and Cryptography https://github.com/topics/blockchain
- Performance benchmarks are posted publicly https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
- It has been load tested doing 100M+ records/day for $10/total costs (CPU, disk, backup) https://www.youtube.com/watch?v=x_WqBuEA7s8
- Explains how consensus, CRDTs, and DAGs work http://gun.js.org/distributed/matters.html
- Except ours is Open Source (MIT/ZLIB/Apache2) and ranked #2 on GitHub in Blockchain and Cryptography https://github.com/topics/blockchain
- Performance benchmarks are posted publicly https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2...
- It has been load tested doing 100M+ records/day for $10/total costs (CPU, disk, backup) https://www.youtube.com/watch?v=x_WqBuEA7s8
- Explains how consensus, CRDTs, and DAGs work http://gun.js.org/distributed/matters.html
Can you please stop spamming this every time a remotely related topic (which apparently now also includes CRDTs and blockchain) is mentioned?
It's probably best to email the mods about things like this, especially if other people have made the same request in the past and been ignored.
GUN is a CRDT, you should know that if you've read my comments from years ago.
I get it, you don't like seeing Open Source (MIT/ZLIB/Apache2) posted about. But that isn't my problem, others do, and I don't profit from it. In fact, it takes a significant amount of my time to help provide free support to any the ~8K developers developers in our community. I'm not ashamed of this, nor do I think it is spam. If I was selling a service or a product, sure. But that isn't the case.
I get it, you don't like seeing Open Source (MIT/ZLIB/Apache2) posted about. But that isn't my problem, others do, and I don't profit from it. In fact, it takes a significant amount of my time to help provide free support to any the ~8K developers developers in our community. I'm not ashamed of this, nor do I think it is spam. If I was selling a service or a product, sure. But that isn't the case.
> I get it, you don't like seeing Open Source (MIT/ZLIB/Apache2) posted about.
No you don't get it. I love hearing about such products - from the people who love using it so much that they feel the need to spread the word around. Submissions about project news are also great, because they can be separately up/downvoted, and don't ride on the coattails of trending submissions. That plus deceitful marketing tricks ("ranked #2 on Github" for a buzzword keyword you picked out, doesn't mean shit) is clearly spam in my book.
> and I don't profit from it
Is it not true that you run a VC-funded company that is behind gun, as you have claimed on other occasions? If it's VC-funded there probably is a profit goal in the foreseeable timeline.
> ~8K developers in our community
Stars on Github doesn't translate to community members, especially if they are gained by high-publicity actions, like posts on HN. This is meant as advice coming from someone who made the mistake of seeing them equivalent in the past. We got ~5k stars via HN for Leaf[0], and I know that only about 10-20 people tried to even use the library. Judging from the activity on the gun repo, it does look to be a little bit better, but not by much.
[0]: https://github.com/autumnai/leaf
No you don't get it. I love hearing about such products - from the people who love using it so much that they feel the need to spread the word around. Submissions about project news are also great, because they can be separately up/downvoted, and don't ride on the coattails of trending submissions. That plus deceitful marketing tricks ("ranked #2 on Github" for a buzzword keyword you picked out, doesn't mean shit) is clearly spam in my book.
> and I don't profit from it
Is it not true that you run a VC-funded company that is behind gun, as you have claimed on other occasions? If it's VC-funded there probably is a profit goal in the foreseeable timeline.
> ~8K developers in our community
Stars on Github doesn't translate to community members, especially if they are gained by high-publicity actions, like posts on HN. This is meant as advice coming from someone who made the mistake of seeing them equivalent in the past. We got ~5k stars via HN for Leaf[0], and I know that only about 10-20 people tried to even use the library. Judging from the activity on the gun repo, it does look to be a little bit better, but not by much.
[0]: https://github.com/autumnai/leaf
The fact that we get so much traffic and contributors to our community from HN means that for every 1 person like you who is so disgruntled and disturbed by other hackers talking about their Open Source projects, that there is a good 70 to 80 who appreciate it, get value from it, discover a new cool tool, and wind up building projects and helping out.
Can you see, at all, from the other perspective? Maybe I annoy the living daylights out of you, but can you also see how you come off as one of those cliche "hater HN commentors"?
If you had 3 worlds to choose from, one where people openly talk about their work which they give away for free to others, or a world where everybody tries to censor, silence, and hate on others for being passionate about their Open Source projects, or a world where, because everybody is so spiteful and nobody is supportive, that everybody is forced to make their software proprietary and charge for it. Which world would you want to be in?
VC funding is another indication/signal of the value I have created. And no, they don't want me to do some stupid "open core" crippleware (like some of my competitors do) or even convert the open source code from MIT/similar to a terrible AGPL/similar license. They want me to create such a core component of infrastructure that we become the defacto standard, like MySQL, Redhat, and others have become.
So no. You have no basis to hate what we are doing, except for your own frustration probably at hate others have given you. Don't pass that along, you are better than that.
Can you see, at all, from the other perspective? Maybe I annoy the living daylights out of you, but can you also see how you come off as one of those cliche "hater HN commentors"?
If you had 3 worlds to choose from, one where people openly talk about their work which they give away for free to others, or a world where everybody tries to censor, silence, and hate on others for being passionate about their Open Source projects, or a world where, because everybody is so spiteful and nobody is supportive, that everybody is forced to make their software proprietary and charge for it. Which world would you want to be in?
VC funding is another indication/signal of the value I have created. And no, they don't want me to do some stupid "open core" crippleware (like some of my competitors do) or even convert the open source code from MIT/similar to a terrible AGPL/similar license. They want me to create such a core component of infrastructure that we become the defacto standard, like MySQL, Redhat, and others have become.
So no. You have no basis to hate what we are doing, except for your own frustration probably at hate others have given you. Don't pass that along, you are better than that.
My remarks have _nothing_ to do with the open source nature of your product.
> that there is a good 70 to 80 who appreciate it, get value from it, discover a new cool tool, and wind up building projects and helping out.
Then why do I always ever see you posting comments about gun, and not any of those people?
I don't argue that HN isn't a place that can also be used for self-promotion, but there is a right time and place for it: "Show HN", and when people specifically post comments seeking out something like what you are building.
There are also mechanisms in place to prevent too much self-promotion, like not being able to submit the same link twice. You know that, as you've also "cleverly" figured out how to circumvent it and submit the gun Github repo >10 times.
> that there is a good 70 to 80 who appreciate it, get value from it, discover a new cool tool, and wind up building projects and helping out.
Then why do I always ever see you posting comments about gun, and not any of those people?
I don't argue that HN isn't a place that can also be used for self-promotion, but there is a right time and place for it: "Show HN", and when people specifically post comments seeking out something like what you are building.
There are also mechanisms in place to prevent too much self-promotion, like not being able to submit the same link twice. You know that, as you've also "cleverly" figured out how to circumvent it and submit the gun Github repo >10 times.
[deleted]
Can you link your project foundations to a peer reviewed paper with adversarial attack analysis?
So no industry/academic peer review yet.
The link is to a self-published whitepaper, and the 'stanford.edu' affiliation is through a visiting postdoc at the Graduate School of Education (nothing CS related).
The link is to a self-published whitepaper, and the 'stanford.edu' affiliation is through a visiting postdoc at the Graduate School of Education (nothing CS related).
The usual complaint is "what about your blockchain crypto economic incentives?" for that, we turned to a PhD in psychology for behavioral economics.
So yes, her PhD is in psychology, and she's a visiting research scientist at Stanford's School of Education.
Meanwhile, I work on the computer science side, although am not an academic (although I interact with plenty of academics, MIT, Berkeley, Stanford, and some European schools). The tech talk I linked to explains how the algorithms work (CRDTs), you don't need to be an academic to understand how it works.
The system has been used on Navy warships, as well. If you'd like to talk to the Sergeant Major who deployed the system in production in the industry, shoot me an email and I'll happily connect you. :)
So yes, her PhD is in psychology, and she's a visiting research scientist at Stanford's School of Education.
Meanwhile, I work on the computer science side, although am not an academic (although I interact with plenty of academics, MIT, Berkeley, Stanford, and some European schools). The tech talk I linked to explains how the algorithms work (CRDTs), you don't need to be an academic to understand how it works.
The system has been used on Navy warships, as well. If you'd like to talk to the Sergeant Major who deployed the system in production in the industry, shoot me an email and I'll happily connect you. :)
Where are the independent security reviews? Also, I don't see past references where you based your new idea.
Please take into account that I am respecting you for your work but this field involves more than a claim. If you look, for example, to simmetric encryption algorithms history, there were a lot of proposals and people involved in trying to crack them. Many of them sounded great but were break by professional cryptographers. The blockchain scene is full of proposals but there are very few security reviews.
Please take into account that I am respecting you for your work but this field involves more than a claim. If you look, for example, to simmetric encryption algorithms history, there were a lot of proposals and people involved in trying to crack them. Many of them sounded great but were break by professional cryptographers. The blockchain scene is full of proposals but there are very few security reviews.
Go have anybody you want look at the code, it is open source. My friends like Dominic Tarr (of Secure Scuttlebutt), security teams at MIT (Christian and Dmitri), and others have consulted with me about it. If you are willing to contribute donations to pay a firm, please do!
You edited your post after I replied.
We're not writing our own cryptography, lol, we're using existing libraries based around OpenSSL using industry accepted and standardized algorithms and curves (ECDSA 256, AES 256, ECDH 256, PBKDF, etc.)
So yes, please email me the moment those get broken/cracked and we'll upgrade to whatever new standards are available.
We're not writing our own cryptography, lol, we're using existing libraries based around OpenSSL using industry accepted and standardized algorithms and curves (ECDSA 256, AES 256, ECDH 256, PBKDF, etc.)
So yes, please email me the moment those get broken/cracked and we'll upgrade to whatever new standards are available.
I hope you understand that you can use secure primitives (such as ECDSA and AES) to build insecure protocols.
And I hope you understand we've spent over a year with a large open source community collectively working on reviewing the implementation, discussing it with colleagues around the world from MIT and Navy personnel and more, working to make it secure, and even now still improving it.
Do you have any vulnerability to report? Please send it in. Security is an always evolving process.
Do you have any vulnerability to report? Please send it in. Security is an always evolving process.
If someone wants to claim a breakthrough it would be simpler to identify current blockchain restrictions and show how they are improved. Once this is clear we can only hope that the relatively few researchers in the field take a look to see if there are obvious inconsistencies.
[1] https://github.com/bft-smart/library
[2] http://repositorio.ul.pt/bitstream/10455/6897/1/TR-2013-07.p...