Filecoin, StorJ and the problem with decentralized storage (2019)(randomoracle.wordpress.com)
randomoracle.wordpress.com
Filecoin, StorJ and the problem with decentralized storage (2019)
https://randomoracle.wordpress.com/2019/12/07/filecoin-storj-and-the-problem-with-decentralized-storage/
122 comments
What do you base AWS storage having "an enormous markup" on? AWS has a type of storage to meet any access need / budget. S3 Glacier Deep Dive for example is perfect for storing archival items and costs $0.00099 per GB-month. So you can store a TB for about $1 per month.
There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data.
There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage.
I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data.
There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage.
I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
> What do you base AWS storage having "an enormous markup" on?
Probably on the basis that backblaze costs an order of magnitude less for the same product.
Probably on the basis that backblaze costs an order of magnitude less for the same product.
AFAIK s3 is multi datacenter whereas B2 is only one.
And you pay S3's markup for every datacenter you want your data to be stored in, plus AWS's network markup for the transfer between those datacenters, do you not?
S3 has lots of options. If you don't want multiple availability zones (which is having multiple copies in multiple datacenters) you can use S3 One Zone-IA.
https://aws.amazon.com/s3/pricing/
https://aws.amazon.com/s3/pricing/
I was replying to a comment which used AWS having multiple datacenters as a benefit over Backblaze which only has one. If your data is only in one datacenter either way, then does it really matter if a particular provider only has one datacenter?
[deleted]
You forgot to mention transfer costs. It is only free within Amazon's VPC. You even pay for transfer between AWS regions (and transfer to CloudFront).
It’s like the people who seriously touted Bitcoin as a cheaper payment mechanism in than credit cards 2013. Few of them explored how ~3% fees are a balance between customer rewards, regulatory priorities, and merchant convenience. In places like Australia regulators made other choices, largely eliminating rewards programs and capping interchange fees at around a tenth of what they are in the US.
There are good arguments for decentralization and censorship resistance, but from a first principles perspective scale and market pressures are good bets for delivering cost efficiencies.
There are good arguments for decentralization and censorship resistance, but from a first principles perspective scale and market pressures are good bets for delivering cost efficiencies.
Clearly the Bitcoin skeptics had the better of the debate ;)
You must also think "scale" and "market pressures" keep down the price of Windows.
Hint: AWS has wonderful margins because everything is tied together, and bandwidth charges to get data out are huge. S3 is a great solution for data used on AWS, but costs more than 30x Storj on bandwidth alone ($90/TB for outside access, plus the cost of doing cross-region replication).
If you're going to work from "first principles" it always helps to have the first hint about how markets actually work--and that different products make different trade-offs (local transfer speed and latency vs. cheap global redundancy)
You must also think "scale" and "market pressures" keep down the price of Windows.
Hint: AWS has wonderful margins because everything is tied together, and bandwidth charges to get data out are huge. S3 is a great solution for data used on AWS, but costs more than 30x Storj on bandwidth alone ($90/TB for outside access, plus the cost of doing cross-region replication).
If you're going to work from "first principles" it always helps to have the first hint about how markets actually work--and that different products make different trade-offs (local transfer speed and latency vs. cheap global redundancy)
I’m not sure I follow. By what metrics were the skeptics wrong about Bitcoin as a viable and cheap transaction platform?
> but costs more than 30x Storj on bandwidth alone ($90/TB for outside access, plus the cost of doing cross-region replication).
Yeah but if use AWS S3 for off site backups, those huge charges cost me exactly $0/TB.
I agree with others - Filecoin and Storj have no meaningful commercial use cases.
Who wants to understand how those things work, let alone use it for anything that involves commercial use cases?
AWS S3: put, get, delete, pay. Done.
Yeah but if use AWS S3 for off site backups, those huge charges cost me exactly $0/TB.
I agree with others - Filecoin and Storj have no meaningful commercial use cases.
Who wants to understand how those things work, let alone use it for anything that involves commercial use cases?
AWS S3: put, get, delete, pay. Done.
Aws bandwidth costs are ridiculous and personally makes public clouds entirely unusable.
Ps everything on aws is marked up by like 300%, it's easy to beat them on cost.
Ps everything on aws is marked up by like 300%, it's easy to beat them on cost.
[deleted](1)
>Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.
Isn't this a non-issue because you can encrypt your data prior to uploading?
Isn't this a non-issue because you can encrypt your data prior to uploading?
It's an issue if the data's public.
> Google scans your Docs for blasphemy against Fauci
What is this referencing? I'm just curious, this is something I haven't heard about.
What is this referencing? I'm just curious, this is something I haven't heard about.
https://reclaimthenet.org/google-drive-takes-down-user-file-...
Discussion: https://news.ycombinator.com/item?id=23275308
As usual, it may not be what you think.
Discussion: https://news.ycombinator.com/item?id=23275308
As usual, it may not be what you think.
I agree that all the concerns can be mitigated. But they're not just theoretical, they're also issues with the current implementations—some of which remain a year and a half after this was written.
For backup scenarios, you can encrypt your data before sending it to Google/Amazon. Filecoin even accommodates (or at least plans to, from what I've read) for nodes that want to store only certain types of content based on legality or whatever.
One important property of decentralized storage is that it prevents data monopolies. For example, if you host your web app on Sia Skynet, then users own their data instead of being locked in by the app developer. It also enables a model where different apps can easily access the same data. The data is in the "cloud", but still owned by the user.
Do you know why Sia is overlooked so often? I see a lot of shills on forums and the founder seems to tweet bitterly about Filecoin, but folks seem optimistic about the tech. So what am I missing that makes Sia not get adoption vs. Filecoin?
The Sia team has let down customers for other tangential projects, and Sia has weaker guarantees than Filecoin. It’s been out a long time now and they have failed to address some core scaling and redundancy concerns. I wouldn’t say it’s bad, but it’s seeming more like maidsafe - never got enough traction and now the interest is on filecoin and arweave.
Filecoin got a lot of funding from prominent investors before they even started. I don't think there's much actual adoption in there.
They spent very little on marketing and focused on building something. That's starting to change now.
Likewise with ScPrime (https://scpri.me) which is like Sia but focused on B2B data storage and is changing to PoS instead of PoW to be more environmentally friendly.
You could implement the same Solid-style model using centralized storage or you could implement lock-in on top of decentralized storage. It's mostly orthogonal.
I think that this paper is more about file storage, not web apps.
And for file storage, there are no data monopolies already. Take for example AWS S3 storage -- there are multiple providers (AWS, Backblaze or local Minio) and many, many clients in all sorts of languages.
And for file storage, there are no data monopolies already. Take for example AWS S3 storage -- there are multiple providers (AWS, Backblaze or local Minio) and many, many clients in all sorts of languages.
I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated:
1. Economies of Scale
2. Network Properties (robustness, scalability, latency, etc.)
It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that.
Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving?
The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates.
At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.
1. Economies of Scale
2. Network Properties (robustness, scalability, latency, etc.)
It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that.
Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving?
The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates.
At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.
Addendum:
I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0].
Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders.
Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user.
/rant
[0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.
I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0].
Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders.
Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user.
/rant
[0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.
I've been keeping my eyes on https://cobox.cloud/ - it's run by some very smart people who have had their brains in the decentralization (not blockchain) space for a while now.
The alpha already lets you do what you're talking about, but I haven't used it yet and so I can't speak to its current stability.
The alpha already lets you do what you're talking about, but I haven't used it yet and so I can't speak to its current stability.
This speaks to the disingenuity of a lot of the decentralization talk in the crypto space. The Beaker browser and the hyperdrive protocol (formerly DAT) is an interesting case study. It has a level of maturity and delivers on many of the ideas that helped filecoin and other token-financed projects raise hundreds of millions of dollars, but it is relatively obscure and I don’t think I know anyone who has used it other than the people I’ve encouraged to check it out. If there were actual demand for dweb technology then it and other projects like it should be getting a lot more attention since it’s a free and relatively user friendly platform compared to services that require managing token exchange risk, accounting, and hassle.
https://beakerbrowser.com/
https://beakerbrowser.com/
a classic essay discussing the real ground truth problems of decentralized anything is by Jo Freeman, called the tyranny of structurelessness [0].
though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what they claim to be, and indeed cannot be.
why? because structureless coordination of groups above a handful members doesn't scale beyond the achievement of the simplest kinds of goals, such as consciousness-raising (or, in crypto-promoter terms, spreading "adoption").
any group objective requiring specialization of labor, coordination of resources and responsibility inevitably succumbs to the insidious creep of informal, but officially disavaowed structural networks of insiders.
it is especially pernicious for newcomers, who have been heavily messaged about "decentralization" or "structurelessness" and engage themselves with the groups identity without any awareness of these cryptic back channel networks that actually govern its operation. only after they are committed do they slowly find out there really is a hierarchy and a command structure.
i highly recommend a quick read for anyone considering the merits of a given groups' claims of being decentralized.
[0] https://www.jofreeman.com/joreen/tyranny.htm
though most of the punches land toward the end, it is a fascinating and short read, in which she dissects, with the sober disappointment of a former optimistic evangelist, exactly why structureless ("decentralized" before that term became vogue) movements are never what they claim to be, and indeed cannot be.
why? because structureless coordination of groups above a handful members doesn't scale beyond the achievement of the simplest kinds of goals, such as consciousness-raising (or, in crypto-promoter terms, spreading "adoption").
any group objective requiring specialization of labor, coordination of resources and responsibility inevitably succumbs to the insidious creep of informal, but officially disavaowed structural networks of insiders.
it is especially pernicious for newcomers, who have been heavily messaged about "decentralization" or "structurelessness" and engage themselves with the groups identity without any awareness of these cryptic back channel networks that actually govern its operation. only after they are committed do they slowly find out there really is a hierarchy and a command structure.
i highly recommend a quick read for anyone considering the merits of a given groups' claims of being decentralized.
[0] https://www.jofreeman.com/joreen/tyranny.htm
I'm familiar with this essay, and agree with most of its conclusions.
However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy!
I'm more interested in organizations that do have some degree of centralization. My hypothetical freelancers are certainly not disorganized. What I'm trying to grasp is the conditions under which any group -- especially one with centralized policymaking -- may seek decentralized infrastructure.
In other words: when does decentralized infrastructure serve the goals of a well-organized group?
However, I am not convinced it is exactly applicable to the situation I have in mind. Freeman's point pertains to extreme decentralization of organizational policy, i.e. the flat hierarchy. These are arguably not even organizations; the essence of organization is precisely hierarchy!
I'm more interested in organizations that do have some degree of centralization. My hypothetical freelancers are certainly not disorganized. What I'm trying to grasp is the conditions under which any group -- especially one with centralized policymaking -- may seek decentralized infrastructure.
In other words: when does decentralized infrastructure serve the goals of a well-organized group?
I believe such infrastructure is better thought of as distributed, rather than decentralized. decentralization is a tendency to resist centralizing forces, it is not an intentional policy of a group. as soon as an identifiable organ in the organization is charged with twiddling the policy knobs that govern control over group resource access, this organ is now distributing this access and it can no longer be considered decentralization.
the paradox is simply this: decentralization implies the absence of a decentralizer. there can be no "center of decentralization."
I contend that the term is much closer to incantation than implementation in practice.
the paradox is simply this: decentralization implies the absence of a decentralizer. there can be no "center of decentralization."
I contend that the term is much closer to incantation than implementation in practice.
> I believe such infrastructure is better thought of as distributed, rather than decentralized.
Hmm, I'm not sure I follow. On a technical level, a pure p2p system continues to function without central coordination even if each individual node is "owned" by the same entity. These seem to be two very different levels of analysis. If I understand your claim correctly, you are pointing out that this hypothetical entity is centralized -- and I agree -- but that doesn't magically turn the p2p system in question into something else.
To reformulate the question: when is it advantageous for this abstract entity to use a p2p system in this way rather than a client-server system?
Is it e.g. a question of scale?
I have a nagging suspicion that it's partly a question of usage-patterns, though I'm struggling to put my finger on it. This is what I mean by "organizational policy" (which is admittedly an opaque term...)
Hmm, I'm not sure I follow. On a technical level, a pure p2p system continues to function without central coordination even if each individual node is "owned" by the same entity. These seem to be two very different levels of analysis. If I understand your claim correctly, you are pointing out that this hypothetical entity is centralized -- and I agree -- but that doesn't magically turn the p2p system in question into something else.
To reformulate the question: when is it advantageous for this abstract entity to use a p2p system in this way rather than a client-server system?
Is it e.g. a question of scale?
I have a nagging suspicion that it's partly a question of usage-patterns, though I'm struggling to put my finger on it. This is what I mean by "organizational policy" (which is admittedly an opaque term...)
I think its a question of how you reason about technology. I see technology as an extension of the agency of those who created and operate it, not as even in theory) an autonomous entity distinct from its users. in this view, who is operating the network is where the decentralization-or-not analysis belongs.
p2p requires peers, so called for good reason, because their interests and cost/benefits must align sufficiently for them to engage in using the same protocol. thus, "who these people are" dominates the question of control over the network.
tl;dr behavior of users networked by a putatively decentralized protocol will ultimately conform to the social organization of those users, not to the rules of the protocol. these will be routed around or backgrounded as needed to achieve the objectives of the dominant members of the group hierarchy.
p2p requires peers, so called for good reason, because their interests and cost/benefits must align sufficiently for them to engage in using the same protocol. thus, "who these people are" dominates the question of control over the network.
tl;dr behavior of users networked by a putatively decentralized protocol will ultimately conform to the social organization of those users, not to the rules of the protocol. these will be routed around or backgrounded as needed to achieve the objectives of the dominant members of the group hierarchy.
Structurelessness and decentralization are only related in that structureless things are decentralized. Decentralized things are rarely structureless. Whether a networking protocol is decentralized or not, it's definitely not structureless; it's nothing but structure, it's a protocol.
Well put. Thanks for putting words on that.
The confusion between structurelessness and decentralization is interesting in and of itself, though. It seems like it is exactly this conflation that prevents people from asking the interesting question: when does a centrally-organized social system (e.g. a business) benefit from a decentralized protocol?
It seems like this should have something to do with usage/access patterns within the business, yet we never hear about this.
The confusion between structurelessness and decentralization is interesting in and of itself, though. It seems like it is exactly this conflation that prevents people from asking the interesting question: when does a centrally-organized social system (e.g. a business) benefit from a decentralized protocol?
It seems like this should have something to do with usage/access patterns within the business, yet we never hear about this.
You can have structure without centralization - see for example federated systems like email. However, the lack of a centralized authority comes with its own issues - see for example federated systems like email.
I won't disagree with your summary of the situation with email, but I think I would add that it is possible to have federated reputation systems which would solve much of the spam problem with email, at least.
To give more detail about the sort of system I mean, here is an excellent description of the approach favoured by the team behind the Matrix communication protocol:
https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix...
To give more detail about the sort of system I mean, here is an excellent description of the approach favoured by the team behind the Matrix communication protocol:
https://matrix.org/blog/2020/10/19/combating-abuse-in-matrix...
Here's an example: Cash decentralizes credit.
"Cash and carry" grocery outlets were a 20th century innovation. [0] Before that, the norm was to have a line of credit with the business, and in many cases, to accept their terms for delivery. Cash transactions anonymize, since settlement is done at the counter. You don't have to assess the buyer, you just need to verify the bills and change are real.
However, that isn't the entire story. While there were businesses using cash before this, they faced difficulties with accounting, supply logistics, and other elements that made it hard to conceive of something like a "supermarket", carrying a vast variety and quantity of goods on a daily basis. So then we have to look at all the pieces that fell into place to make it possible.
The automobile decentralized access to mobility, making carry-away a real possibility for more people, and thus making it possible to apply cash-and-carry in more places. Supporting elements like cash registers and refrigeration were becoming mature enough to support new forms of retail and allow more parts of the transaction to be delegated to local outlets and low-wage employees. The inter-war years really saw a whole set of technological innovations that were used in combinations to propel social changes and different categories of business(e.g. fast food), many of them decentralized in some respects but centralized in others - supermarket chains, as opposed to local markets.
These are the kinds of changes that are hardest to assess in full; when you decentralize one thing, centralization is "squeezed" into other parts of the economy, it seems. The obvious example for this phenomenon is Amazon, leveraging an apparently decentralizing mechanism(online retail - premised on an internet with sufficient bandwidth and security to list goods and take payments) into becoming the world's largest retailer. So it's centralized on one axis, but decentralized on others - a buyer no longer has to go to a particular physical location to purchase something, when all of it can be delivered to the doorstep.
[0] https://en.wikipedia.org/wiki/Cash_and_carry_(wholesale)
"Cash and carry" grocery outlets were a 20th century innovation. [0] Before that, the norm was to have a line of credit with the business, and in many cases, to accept their terms for delivery. Cash transactions anonymize, since settlement is done at the counter. You don't have to assess the buyer, you just need to verify the bills and change are real.
However, that isn't the entire story. While there were businesses using cash before this, they faced difficulties with accounting, supply logistics, and other elements that made it hard to conceive of something like a "supermarket", carrying a vast variety and quantity of goods on a daily basis. So then we have to look at all the pieces that fell into place to make it possible.
The automobile decentralized access to mobility, making carry-away a real possibility for more people, and thus making it possible to apply cash-and-carry in more places. Supporting elements like cash registers and refrigeration were becoming mature enough to support new forms of retail and allow more parts of the transaction to be delegated to local outlets and low-wage employees. The inter-war years really saw a whole set of technological innovations that were used in combinations to propel social changes and different categories of business(e.g. fast food), many of them decentralized in some respects but centralized in others - supermarket chains, as opposed to local markets.
These are the kinds of changes that are hardest to assess in full; when you decentralize one thing, centralization is "squeezed" into other parts of the economy, it seems. The obvious example for this phenomenon is Amazon, leveraging an apparently decentralizing mechanism(online retail - premised on an internet with sufficient bandwidth and security to list goods and take payments) into becoming the world's largest retailer. So it's centralized on one axis, but decentralized on others - a buyer no longer has to go to a particular physical location to purchase something, when all of it can be delivered to the doorstep.
[0] https://en.wikipedia.org/wiki/Cash_and_carry_(wholesale)
As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it.
Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon.
> knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post.
This is something I'd love to hear more about.
Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon.
> knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post.
This is something I'd love to hear more about.
Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer.
Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
> dozens of independent entities
or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future"
proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one.
nothing but faith-based decentralization here, as with all cryptocurrencies.
or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future"
proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one.
nothing but faith-based decentralization here, as with all cryptocurrencies.
The argument (of the second post) is that their behavior will be correlated because they all have (more or less) the same incentives and cost structure.
> SLAs -- something that a blockchain can't really offer.
Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely.
If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties.
The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely.
If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties.
The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
I think this speaks to the difference between what SLAs really are and what people wishfully think they are. People think an uptime SLA means "we will not go down much" but it really means "one day we will go down hard and we'll give you a refund (which will be insignificantly small compared to the pain we caused you)".
In the case of Storj, I believe that the redundancy between storage nodes is sufficient that something like only 20 out of 80 nodes need to respond in order to restore your data.
[deleted]
S3 has no durability SLA, only an availability SLA. S3 also explicitly says it is not designed for 100% durability, but for a certian number of nines (99.999999999%).
Blockchain has neither a durability or availability SLA.
I'm not sure that comparison makes much sense, it's like saying that HTTP has no SLA. AFAIU, Filecoin miners are free to offer SLAs for storage, and clients are free to enter into deals only with miners who offer such SLAs.
An SLA just gives you the right to fee reductions or potential compensation for downtime. And not all SLAs are created equal. Meanwhile, for example, many blockchains such as Bitcoin and Ethereum, have 100% uptime. Not 99.99%...100%.
"uptime" is such a weird concept when talking about decentralized networks as they are local-first and also build with many peers. And which "uptime" is being referred to here, read or write? If it's write, fees have definitely spiked at points to make it mostly useless for most people "using" it, but you can technically still write to it. And read is just always available as long as you have a copy of the data somewhere.
Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
That's pretty hand-wavy. Anyone can read the published code, but you can't read the code being run by each individual entity in the network. Any individual might change their server to not make the stored data available to those paying for the storage.
It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system.
Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system.
Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
In fairness, real-company SLAs also tend to be hand-wavy. People like to pretend it's "they'll never be down more than X" or "if they're ever down more than X they'll give us a refund", but it's usually more like "the service provider said that it didn't count because only one part of their service was 'degraded', and after we threatened to take them to court they threw us a $13 discount'".
What if the system had 2 kinds of nodes, storage nodes, and then verification nodes. The storage nodes, store the data. The verification nodes, hit the storage nodes and verify the data is there and available. If the storage node fails a verification attempt, the storage node pays a fee. After a certain amount of failures, it is banned from the network.
Disclaimer: I have no idea how these systems currently work.
Disclaimer: I have no idea how these systems currently work.
This is basically how decentralized storage works except you can't really ban anyone so a storage provider will lose their deposit if they fail verification. As the article says, this forces storage providers to lock up capital and even that is no guarantee since there may be situations where losing the deposit is actually cheaper than staying in business.
I can run the code behind a network that prevents outbound file transfers for large data.
So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
And again, even if we restrict ourselves to honest parties, hardware failures happen. Auditing the code tells you nothing about their infrastructure.
I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.
Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look.
It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
Extremely expensive in terms of storage, but still much cheaper than S3.
I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
[deleted]
The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.
Why is Sia overlooked every single time this topic is brought up? It is shocking to me that Filecoin, an example of an ICO darling with no working product, is always at the top of the list.
I have used Sia and Skynet to build several useful decentralized media sharing applications with zero loss of data for over a year.
Sia currently costs 8$ TB/month, storj costs a fixed forever* 4$ TB/month.
* not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.
* not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.
Sia has been operating at a large scale for years and has a running decentralized CDN service (Skynet).
Just for everybody to be in the same page.
Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol.
From ipfs.io about Blockchain
>With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain.
Please correct me if I am wrong.
Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol.
From ipfs.io about Blockchain
>With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain.
Please correct me if I am wrong.
You are corrent; I think the article is just clumsily worded, though. I think what he's saying is that IPFS gives you references to immutable data, which you are then free to commit to your favorite blockchain.
Interesting article. While reading about Filecoin the first time, I had to stop every paragraph and think "but what if the node just doesn't give you your data back when you finally ask for it?". And as pointed out by part 2 of the article, not only can they choose not to give it back, but they are incentivized not to.
edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?
edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?
As someone who's been interested in filecoin mining for a while, my main question is around legal liability for miners. For instance, if I am storing other people's data and it turns out someone was storing something illegal (unbeknownst to me), could I be held liable under US law?
This is where the DMCA and CDA safe harbors help you.
Why do you think this is different than hosting a normal website with file uploads? Literally millions of people have made such websites (especially around 2000-2010).
That’s true but when you host a website, you can decide what types of files can be uploaded. With this sort of decentralized storage network, people will include whatever files they want. And to put a finer point on it, no doubt some significant portion of the file content will be horrifying video content, some of which will be illegal. My question is more around the liability for that, even if it is encrypted and the miner doesn’t know what it is.
A small filecoin operation would not have the legal funds to defend against an aggressive prosecutor trying to “take down a network of dangerous pornographers” and would get little sympathy from a general public that doesn’t understand the tech.
Until there is strong legal precedent, I wouldn’t entertain a mining operation. Maybe I’m paranoid but it just doesn’t seem worth the risk.
A small filecoin operation would not have the legal funds to defend against an aggressive prosecutor trying to “take down a network of dangerous pornographers” and would get little sympathy from a general public that doesn’t understand the tech.
Until there is strong legal precedent, I wouldn’t entertain a mining operation. Maybe I’m paranoid but it just doesn’t seem worth the risk.
It's infeasible to make sure a server is only accepting "legal" content. I've never heard of a case of someone being jailed because his system involuntarily received some data that happens to be illegal. Normally what happens if there is illegal data that anyone cares about, the hoster is contacted and told to remove it.
>it just doesn’t seem worth the risk.
Well, the concern is only yours.
This does raise the point that if police contact you to remove some data you are hosting for someone, you lose money, and they will likely not give you back whatever money you lost (all the more reason it should be fully anonymous like Freenet but IPFS and co are obsessed with trying to solve every use case in the world including legal compliance).
>it just doesn’t seem worth the risk.
Well, the concern is only yours.
This does raise the point that if police contact you to remove some data you are hosting for someone, you lose money, and they will likely not give you back whatever money you lost (all the more reason it should be fully anonymous like Freenet but IPFS and co are obsessed with trying to solve every use case in the world including legal compliance).
IPFS is mostly fine technology for storing data in a content-addressable way.
FileCoin is, at best, an inefficient way to incentivize people to store your data in an imaginary trustless world. (At worst, it's an obvious scam.)
You can imagine any hosting provider offering regular IPFS hosting. Sign up on our website, send us a list of data to store, pay us money, and we'll guarantee you can get it back or you can sue us - same as the guarantees by any conventional hosting provider. This works fine and well without gluing Filecoin to it.
FileCoin is, at best, an inefficient way to incentivize people to store your data in an imaginary trustless world. (At worst, it's an obvious scam.)
You can imagine any hosting provider offering regular IPFS hosting. Sign up on our website, send us a list of data to store, pay us money, and we'll guarantee you can get it back or you can sue us - same as the guarantees by any conventional hosting provider. This works fine and well without gluing Filecoin to it.
The advantage would be to have a program on my computer that would negotiate prices with providers and manage latency and redundancy. This could of course also be solved if storage providers, as a group, standardized on a single protocol (creating a commodity industry.)
I'm not sure there's a huge difference between that and this, other than anonymity for providers (which is the major critique of this blogpost.)
I'm not sure there's a huge difference between that and this, other than anonymity for providers (which is the major critique of this blogpost.)
IPFS is not a storage technology. IPFS is a retrieval technology.
In fact the two technologies would complement each other really well. Imagine that you could pay someone to make content available on IPFS, basically a distributed pinning service. However this isn't trivial because IPFS generally assumes that people with files would like to share them, however a for-profit storage node would be incentivized to use as little bandwidth as possible.
In fact the two technologies would complement each other really well. Imagine that you could pay someone to make content available on IPFS, basically a distributed pinning service. However this isn't trivial because IPFS generally assumes that people with files would like to share them, however a for-profit storage node would be incentivized to use as little bandwidth as possible.
IPFS includes both storage and retrieval.
I outlined in my post a specific, trivial way to pay somebody to make data available on IPFS.
I outlined in my post a specific, trivial way to pay somebody to make data available on IPFS.
I guess we are arguing semantics but your last paragraph sounds like you are asking someone to do storage and make it available over IPFS, I still see the storage part as quite disparate.
This is like saying HTTP provides storage because you can pay someone to host your files and allow you to access them over HTTP.
This is like saying HTTP provides storage because you can pay someone to host your files and allow you to access them over HTTP.
> Cloud services are ruled by a ruthless economy of scales. This is where Amazon, Google, MSFT and a host of other cloud providers shine, reaping the benefits of investment in data-centers and petabytes of storage capacity.
I dunno about that. Amazon and Google, at least, charge significantly more than I could go out and buy a few drives for on sale. (AND, of course, they charge it perpetually. Dunno how StorJ works.)
I dunno about that. Amazon and Google, at least, charge significantly more than I could go out and buy a few drives for on sale. (AND, of course, they charge it perpetually. Dunno how StorJ works.)
Economies of scale in the cloud have one thing going for them: massive reliability.
As someone responsible for my company's data, I cannot make a sound argument to convince management (or myself) to use anything but Amazon S3 (or Google or Microsoft) cloud. The data is simply too important to trust to a smaller entity.
Maybe coin storage can boost Storj's reputation. And I certainly favor decentralization for Crypto.
As someone responsible for my company's data, I cannot make a sound argument to convince management (or myself) to use anything but Amazon S3 (or Google or Microsoft) cloud. The data is simply too important to trust to a smaller entity.
Maybe coin storage can boost Storj's reputation. And I certainly favor decentralization for Crypto.
So granted the durability and availability is better than centralized solutions, you'd go with the distributed/decentralized solution?
If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's API that abstract that away, introducing more error surface) so you can also store the data in multiple places with the same API.
Or, you can just use a open source decentralized solutions and run it locally geographically, hosted on dedicated servers and "in the cloud" (meaning other users who host it) with the same API without really caring about _where_ the files are, just _which_ files you need.
And that's the beauty of content-addressable networks like many of these are. It doesn't really matter where the data is as long as you know what data you need. That makes it easier to provide reliability for yourself, not harder.
Once you dig down in the details and compare them both carefully, I think you can make sound arguments both ways, it's not as black & white as you make it. Ultimately it depends on one's use case.
If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's API that abstract that away, introducing more error surface) so you can also store the data in multiple places with the same API.
Or, you can just use a open source decentralized solutions and run it locally geographically, hosted on dedicated servers and "in the cloud" (meaning other users who host it) with the same API without really caring about _where_ the files are, just _which_ files you need.
And that's the beauty of content-addressable networks like many of these are. It doesn't really matter where the data is as long as you know what data you need. That makes it easier to provide reliability for yourself, not harder.
Once you dig down in the details and compare them both carefully, I think you can make sound arguments both ways, it's not as black & white as you make it. Ultimately it depends on one's use case.
I don't see how this addresses the reliability claim. There's a big difference between explaining that the data is stored in a Google cloud with certified 11-nines reliability, versus stored on a N number of machines in multiple clouds with unknown specs.
How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliability is accounted for by the high-level service?
How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliability is accounted for by the high-level service?
You'd say something like "This content is hosted by 100 providers, with each having individual average uptime of 99.9%"
I see. But how do I verify that 99.9%? Aren't the distributed services anonymous?
If you're willing to assume that each invididual server has some level of reliability (e.g. "one nine") and you assume failures are uncorrelated you can build a more reliable system using erasure coding and reason mathematically about the resulting reliability. http://www.suaybarslan.com/Reliability_Systems_14.pdf
Hi! CTO of Storj here. This article makes some reasonable points, and it makes some unreasonable ones.
First, on the reasonable: absolutely, privacy and security should be layered on top of whatever storage platform you use using your own encryption. No objection! That's the right approach. It is always better to bring your own encryption, and having it layered so your storage provider can't have access is great. I have much love for @cperciva's Tarsnap. It's awesome.
The downside with a product that punts encryption to the user is that users /typically don't do this/. In fact, users like to share things and serve content to others. Bringing your own encryption that the sharing infrastructure doesn't understand means it is much harder to actually use the product for sharing. This is why Storj includes hierarchical encryption for delegated sharing. It's not that embedding encryption into the framework is better encryption by any means, but it is a better default. Users are by default protected by their own keys that we don't have access to, which is a better default than the cloud. Should people use something else also? Sure! If it suits them.
On the unreasonable:
The author says:
> [I]t is very unlikely that a decentralized storage market can offer an alternative that can compete against centralized providers— AWS, Google, Azure— when measured on any of these [cost, speed, reliability] dimensions.
This is a testable assumption, and the author didn't test it (post is from December of 2019, we were in late beta then, working very well).
Storj is cheaper ($4/TB/mo, was $10/TB/mo in December of 2019), provides 11 9s of durability (no SLA offered until March 2020, but we had and still have never lost a single object out of billions), and is equivalent in speed to providers like Backblaze (and getting faster). It's also multiregion by default (obviously). This is a screaming deal on all dimensions. Try it out for yourself. Our metrics are delivering on all three dimensions.
This author is picking and choosing between architectures of Filecoin and Storj, which is fine, but each should be evaluated in isolation. Filecoin and Storj make very different design decisions with very different payoffs. The author argues Storj can't deliver on its promises with appeals to Filecoin's architecture.
Filecoin absolutely uses blockchains and proof of storage and whatever else. In fact, Filecoin does fall victim to requiring lots of resources (see https://docs.filecoin.io/mine/hardware-requirements/).
But this is just true of Filecoin, this isn't a problem with decentralized storage in general, and Storj is an excellent counter example. Storj does not require lots of resources (and in fact many of our node operators use Raspberry Pis). Storj is not a blockchain, uses statistical audits that are low-effort and low-CPU for storage node operators, and works great for idle capacity.
If you're interested in how we can achieve these things, you might take a look at an older blog post I wrote explaining (without naming) why Filecoin's architecture is fundamentally flawed: https://www.storj.io/blog/replication-is-bad-for-decentraliz...
Storj and Filecoin are both decentralized storage products, but they are really fundamentally very different, and their differences are worth understanding.
First, on the reasonable: absolutely, privacy and security should be layered on top of whatever storage platform you use using your own encryption. No objection! That's the right approach. It is always better to bring your own encryption, and having it layered so your storage provider can't have access is great. I have much love for @cperciva's Tarsnap. It's awesome.
The downside with a product that punts encryption to the user is that users /typically don't do this/. In fact, users like to share things and serve content to others. Bringing your own encryption that the sharing infrastructure doesn't understand means it is much harder to actually use the product for sharing. This is why Storj includes hierarchical encryption for delegated sharing. It's not that embedding encryption into the framework is better encryption by any means, but it is a better default. Users are by default protected by their own keys that we don't have access to, which is a better default than the cloud. Should people use something else also? Sure! If it suits them.
On the unreasonable:
The author says:
> [I]t is very unlikely that a decentralized storage market can offer an alternative that can compete against centralized providers— AWS, Google, Azure— when measured on any of these [cost, speed, reliability] dimensions.
This is a testable assumption, and the author didn't test it (post is from December of 2019, we were in late beta then, working very well).
Storj is cheaper ($4/TB/mo, was $10/TB/mo in December of 2019), provides 11 9s of durability (no SLA offered until March 2020, but we had and still have never lost a single object out of billions), and is equivalent in speed to providers like Backblaze (and getting faster). It's also multiregion by default (obviously). This is a screaming deal on all dimensions. Try it out for yourself. Our metrics are delivering on all three dimensions.
This author is picking and choosing between architectures of Filecoin and Storj, which is fine, but each should be evaluated in isolation. Filecoin and Storj make very different design decisions with very different payoffs. The author argues Storj can't deliver on its promises with appeals to Filecoin's architecture.
Filecoin absolutely uses blockchains and proof of storage and whatever else. In fact, Filecoin does fall victim to requiring lots of resources (see https://docs.filecoin.io/mine/hardware-requirements/).
But this is just true of Filecoin, this isn't a problem with decentralized storage in general, and Storj is an excellent counter example. Storj does not require lots of resources (and in fact many of our node operators use Raspberry Pis). Storj is not a blockchain, uses statistical audits that are low-effort and low-CPU for storage node operators, and works great for idle capacity.
If you're interested in how we can achieve these things, you might take a look at an older blog post I wrote explaining (without naming) why Filecoin's architecture is fundamentally flawed: https://www.storj.io/blog/replication-is-bad-for-decentraliz...
Storj and Filecoin are both decentralized storage products, but they are really fundamentally very different, and their differences are worth understanding.
Leaving aside imperfections of each, for user with extra bandwidth and storage, which one offers better path for monetization?
[deleted]
The article raises a question I've wondered about before: how do you know that anyone will transfer you your data when you need it? You can require people to prove that they stored it, but how can you require people to transfer it out to you?
You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?
You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?
The answer is incentives: if you're paying to retrieve data, why shouldn't they take your business?
But do they send you the data first, or do you send them the payment first?
If they send you the data first, then why should you pay them? If you send them the payment first, then why should they send it?
I'm sure there's a piece I'm missing, but have I missed something obvious?
If they send you the data first, then why should you pay them? If you send them the payment first, then why should they send it?
I'm sure there's a piece I'm missing, but have I missed something obvious?
Data retrieval is broken into blocks so it's a repeated game with incentives to cooperate.
AFAIK Filecoin uses payment channels so you probably lock some money, retrieve data, then unlock the payment.
AFAIK Filecoin uses payment channels so you probably lock some money, retrieve data, then unlock the payment.
Incremental transfer seems like it should give an incentive to cooperate, but I don't think so?
If I pay you upfront for the first block, what stops you from then saying: actually I still want more money to start transfering the first block?
If I pay you upfront for the first block, what stops you from then saying: actually I still want more money to start transfering the first block?
But if the token price rises then the node can just delete your data and make a new deal with someone else and get more profit.
One general solution is to have persistent nodes in the network that develop reputation over a duration of time, who perform operations that can't be done entirely trustlessly. In Storj's case these are called Satellites.
> Whenever a Satellite on the Storj network has a less than stellar payment, demand generation, or performance history, there is a strong incentive for the storage nodes to avoid accepting its data. When a new Satellite joins the network, the participating storage nodes will commence their own vetting process. This process limits their exposure to the new and unknown Satellite, while building trust over time to highlight which of the Satellites have the best payment record.
They also incentivize complete file delivery by paying for chunks of bandwidth as they're used to deliver the file, rather than all up front.
https://www.storj.io/storj.pdf
> Whenever a Satellite on the Storj network has a less than stellar payment, demand generation, or performance history, there is a strong incentive for the storage nodes to avoid accepting its data. When a new Satellite joins the network, the participating storage nodes will commence their own vetting process. This process limits their exposure to the new and unknown Satellite, while building trust over time to highlight which of the Satellites have the best payment record.
They also incentivize complete file delivery by paying for chunks of bandwidth as they're used to deliver the file, rather than all up front.
https://www.storj.io/storj.pdf
[deleted]
> Another option is requiring service providers to put up a surety bond. Before they are allow to participate in the ecosystem, they must set aside a lump sum on the blockchain held in escrow. These funds would be used to compensate any customers harmed by failure to honor storage contracts. But this has the effect of creating additional barriers to entry and locking away capital in a very unproductive way.
If bonds solve the bad actor problem, then this use of capital would directly enable safe data storage. I guess it depends on your definition of “productive”, but that definitely feels like a good use of capital to me.
If bonds solve the bad actor problem, then this use of capital would directly enable safe data storage. I guess it depends on your definition of “productive”, but that definitely feels like a good use of capital to me.
Article looks like it is from December 2019. Shouldn't (2019) be in the title?
Is there any sensible study on those system's energy consumption?
Given that public clouds are ruthlessly optimized for efficiency and they don't have to perform trustless proof of anything, it's virtually certain that decentralized storage is less efficient in every way than centralized.
What does "virtually certain" mean here anyway? And on what scale are you imagining things?
Let's say the entirety of YouTube was decentralized in a way that every internet connected device plugged in to the wall starts sharing watched content to other viewers who want to watch that content. If it's your neighbor, they'll download it straight from you, via the closest router/hub. Suddenly the company's hosting is more providing a "core archive" of data, and every user becomes a edge node of that content. Internet providers can, if they chose, pre-share content they think will be popular, in order to speed up their own infrastructure, and so on.
I'm not saying this is bullet proof or anything, I just think you're too quick to dismiss it without really saying why you think it's impossible (or "virtually certain").
Let's say the entirety of YouTube was decentralized in a way that every internet connected device plugged in to the wall starts sharing watched content to other viewers who want to watch that content. If it's your neighbor, they'll download it straight from you, via the closest router/hub. Suddenly the company's hosting is more providing a "core archive" of data, and every user becomes a edge node of that content. Internet providers can, if they chose, pre-share content they think will be popular, in order to speed up their own infrastructure, and so on.
I'm not saying this is bullet proof or anything, I just think you're too quick to dismiss it without really saying why you think it's impossible (or "virtually certain").
I love P2P and I have been into it since the Napster days but it isn't feasible any more. The value of a random Internet device is actually negative because the complexity of wrangling it outweighs what little resources it could contribute. (See Spotify and Joost for examples.)
I assume all these decentralized storage systems are datacenter-based so I'm comparing random Chinese datacenters vs. FAANG datacenters.
I assume all these decentralized storage systems are datacenter-based so I'm comparing random Chinese datacenters vs. FAANG datacenters.
Youtube (and most other streaming services) is already decentralized into thousands of edge caches. Decentralizing it further doesn't make much sense because the network cost is about the same if individual homes or their nearest ISP DC has the cache.
Most home networks are wireless only at this point which is pretty bad for throughput, latency, and reliability if serving other homes. Maybe ISPs could start building some storage and IPFS into modems? I think handling the metadata of a p2p system might still be too costly. Imagine every neighborhood having to run a torrent tracker for every media file produced in the last ~year.
Most home networks are wireless only at this point which is pretty bad for throughput, latency, and reliability if serving other homes. Maybe ISPs could start building some storage and IPFS into modems? I think handling the metadata of a p2p system might still be too costly. Imagine every neighborhood having to run a torrent tracker for every media file produced in the last ~year.
Since this was written, Filecoin is up about 10,000% or something stupid.
If you pay attention to the naysayers, you're gonna have fun staying poor.
If you pay attention to the naysayers, you're gonna have fun staying poor.
A better solution would be payment based on a minimum GB and bandwidth where the data is constantly being streamed back over the network with a payout period of ~5min. This way you can guarantee that you will get all your data back at least once a week. You can also guarantee via network latency that your data is not being collocated. Sounds crazy but it would be more energy efficient than bitcoin.
[deleted]
You can use the data field in etheruem for decentralized and permaenent storage (convert the file to hex ad then paste it) but very, very expensive .
Not with middle-out compression.
The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate nodes to show X nines of pool reliability.
Is it a bad idea to use this as your only source of storage? Of course. Is it less useful than AWS if you do large local burst queries? Of course.
But none of the concerns mentioned in the post are relevant. This can be cheaper than AWS because they charge an enormous markup, and this automatically provides worldwide replication and accessibility.
Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.