Decentralising the web: Why is it so hard to achieve?(computing.co.uk)
computing.co.uk
Decentralising the web: Why is it so hard to achieve?
https://www.computing.co.uk/ctg/news/3036546/decentralising-the-web-why-is-it-so-hard-to-achieve
82 comments
>What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?
Well forcing those companies to reform is simply impossible. That's because their incentives are all wrong, so they will always slither around anything you try to get them to do right.
That means that decentralization, as hard as it might be, is always vastly easier than the alternative.
Well forcing those companies to reform is simply impossible. That's because their incentives are all wrong, so they will always slither around anything you try to get them to do right.
That means that decentralization, as hard as it might be, is always vastly easier than the alternative.
> What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?
Those companies have proved they can't be trusted.
So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
Those companies have proved they can't be trusted.
So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
Centralized services use specially taylored engagement mechanics for people to get addicted to their stimuli.
They do so because there is an economic incentive for them.
Decentralized will not do this, as long as they remain non-profit and strive to stay ethical.
This means that the average user will prefer the unethical and addictive platform. Same thing as with fast-food. Not only do decentralized systems need to be technically competitive and surpass hurdles that centralized systems do not have, but they also need users to be made aware of those issues.
They are necessary but utopic.
They do so because there is an economic incentive for them.
Decentralized will not do this, as long as they remain non-profit and strive to stay ethical.
This means that the average user will prefer the unethical and addictive platform. Same thing as with fast-food. Not only do decentralized systems need to be technically competitive and surpass hurdles that centralized systems do not have, but they also need users to be made aware of those issues.
They are necessary but utopic.
I disagree. The "average" user nowadays knows or at least has heard of the problems with fast food. There have been numerous campaigns about eating healthy and as many people who have done the transformation from an unhealthy full of junk food diet to a healthy one with exercising can attest the benefits are there.
We need similar campaigns for an ethical internet. People need to understand the problems with the current options and be offered alternatives as good as possible. At the end of the day we can't force companies to behave "ethically"( whatever that means depending on your definition) so a decentralized options will be a better one.
We need similar campaigns for an ethical internet. People need to understand the problems with the current options and be offered alternatives as good as possible. At the end of the day we can't force companies to behave "ethically"( whatever that means depending on your definition) so a decentralized options will be a better one.
> There have been numerous campaigns about eating healthy and as many people who have done the transformation from an unhealthy full of junk food diet to a healthy one with exercising can attest the benefits are there.
And yet america is fatter than ever and getting fatter quicker than ever.
I think it's time we accept that most people are sheep that give in to even the most primitive forms of manipulation, even after being made aware of it.
Or to put it a bit more delicately, our brain is hardwired to optimise for short term reward functions and the vast majority of us are unable to seek longer term rewards even when consciously desiring them.
And yet america is fatter than ever and getting fatter quicker than ever.
I think it's time we accept that most people are sheep that give in to even the most primitive forms of manipulation, even after being made aware of it.
Or to put it a bit more delicately, our brain is hardwired to optimise for short term reward functions and the vast majority of us are unable to seek longer term rewards even when consciously desiring them.
>I think it's time we accept that most people are sheep that give in to even the most primitive forms of manipulation, even after being made aware of it.
So what? That doesn't mean that the campaigns don't work. There has been a decent number of people changing their lives for the better. The fact that there is a tremendous amount of variation on cultural and education levels in the States simply means that certain areas might lag behind in changes. That doesn't mean the changes won't eventually happen.
So what? That doesn't mean that the campaigns don't work. There has been a decent number of people changing their lives for the better. The fact that there is a tremendous amount of variation on cultural and education levels in the States simply means that certain areas might lag behind in changes. That doesn't mean the changes won't eventually happen.
> So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
The problem is that decentralized systems are always going to be less usable than the centralized systems they compete with. Centralization is what makes better usability possible.
The problem is that decentralized systems are always going to be less usable than the centralized systems they compete with. Centralization is what makes better usability possible.
> Centralization is what makes better usability possible.
I disagree with this premise.
There's less opportunity for monetary gain, so less effort has been invested in decentralized systems.
I don't think it's an actual inherent technical limitation of decentralized systems. In fact decentralized systems promise many technical benefits. Of course there are also challenges, but in my opinion, not insurmountable ones.
I disagree with this premise.
There's less opportunity for monetary gain, so less effort has been invested in decentralized systems.
I don't think it's an actual inherent technical limitation of decentralized systems. In fact decentralized systems promise many technical benefits. Of course there are also challenges, but in my opinion, not insurmountable ones.
There's a kernel of truth in that premise, in that centralization enables better usability by making things within a centralized system consistent, and thus interoperable.
When you let people run away with their creative freedom, you get the Web. That is, nothing talks to anything else in any reasonable way.
When you let people run away with their creative freedom, you get the Web. That is, nothing talks to anything else in any reasonable way.
I don't think so. Standardisation is what makes decentralisation possible. The only one thing that the likes of Facebook really do well compared to decentralised is spam detection.
And by the way - spam is a problem that those platforms don't solve themselves either as they use their communities to filter that out. So it's only a matter of time before those communities realise they might as well work outside those platforms.
And by the way - spam is a problem that those platforms don't solve themselves either as they use their communities to filter that out. So it's only a matter of time before those communities realise they might as well work outside those platforms.
I would think the main problem is centralized services can have benefits that make them superior even when done unethically ( or perhaps quasi-ethitically ).
Building a new internet.
Back in the old days of the internet, universities and government agencies developed the algorithms.
Companies just provided the hardware.
Perhaps we need to go back to that model.
Companies just provided the hardware.
Perhaps we need to go back to that model.
100% agree as far as ethics are concerned. But then you get maintenance costs( both physical and software) and support. Simply put companies can do this better than government agencies( and more on time wrt patching for example). Can you imagine a gov agency responding to a meltdown like exploit?
Most of the systems they developed weren't very good and have been out competed. Going back to that doesn't seem like a solution to anything. How many people prefer IRC to Slack?
> But people only want to use centralized services.
That's not true. "People" don't care whether the service is centralised or decentralised. They just want the practical benefits of something that looks like a centralised service.
That's not true. "People" don't care whether the service is centralised or decentralised. They just want the practical benefits of something that looks like a centralised service.
Building a new internet, of course.
Humanity had built one in just a few decades. Yet, it hadn't managed to successfully enforce any particular human or group behavior in a few millenia. I'm not even really sure humanity has uniform idea about what honesty and transparency actually mean and whenever and how they should be enforced.
Humanity had built one in just a few decades. Yet, it hadn't managed to successfully enforce any particular human or group behavior in a few millenia. I'm not even really sure humanity has uniform idea about what honesty and transparency actually mean and whenever and how they should be enforced.
What that says is that most of the content distribution networks which existed outside centralised systems required a good amount of mental workload for the operator. Back then we did not have the same culture of automated DevOps and so they perished.
I don't think centralisation per se is a defining feature of the platforms you're speaking about.
I don't think centralisation per se is a defining feature of the platforms you're speaking about.
The "user owned co-operative" model for these things is extremely under-explored. However I wonder if it only works for smaller communities; certainly it wouldn't give the same rapid global scaling that Facebook had.
Maybe not?
Last week I gave a talk on the history of decentralisation, or at least my history with it. One of my examples of a successful case of decentralisation is Minecraft vs Lego Universe. Both quite similar games, but Minecraft totally crushed Lego and out-scaled it to an insane degree. As far as I can tell this is largely because Minecraft was much more decentralised (mods, user run servers). In particular Lego Universe was crippled by the cost of centralised content moderation that Mojang simply didn't have at all.
There are examples of decentralised user-owned cooperative type systems outscaling and outperforming centralised alternatives. I'm uncertain to what extent you can make sweeping generalisations ... there certainly seem to be more cases of successful centralised platforms than decentralised, but this is partly because we tend to view decentralisation as an absolute rather than as a spectrum, and partly because a few companies with very strong hiring pipelines have subsidised the creation of vast quantities of centralised software.
Last week I gave a talk on the history of decentralisation, or at least my history with it. One of my examples of a successful case of decentralisation is Minecraft vs Lego Universe. Both quite similar games, but Minecraft totally crushed Lego and out-scaled it to an insane degree. As far as I can tell this is largely because Minecraft was much more decentralised (mods, user run servers). In particular Lego Universe was crippled by the cost of centralised content moderation that Mojang simply didn't have at all.
There are examples of decentralised user-owned cooperative type systems outscaling and outperforming centralised alternatives. I'm uncertain to what extent you can make sweeping generalisations ... there certainly seem to be more cases of successful centralised platforms than decentralised, but this is partly because we tend to view decentralisation as an absolute rather than as a spectrum, and partly because a few companies with very strong hiring pipelines have subsidised the creation of vast quantities of centralised software.
Great example! I would like to check out your talk if you have a recording? I'm currently working on some plugins for https://www.openmined.org to connect to Unity and Minecraft to create local minecraft servers that are fed sensor data from the real world that triggers actions within a minecraft model of the players local environment. I am fascinated by the AR layers that are possible to connect game play to real world collective action.
> We just need to come up with better ways of funding them that don’t involve building a massive survellaince apparatus.
You're absolutely, completely, 100% right. This is a critical need!
Sad to say, I can think of several models that have been tried - donations, subscriptions, and merchandising come to mind. None require building a subscription apparatus! This makes them morally superior in very important ways, but all of those approaches come with distinct downsides. They all have distortionary effects on the user experience and condition the continued existence on the service on a handful of people.
Perhaps we should stop and think carefully about why surveillence-fuelled ads are so popular an approach. It's a systems that lets one monetize lurkers who are otherwise minimally interactive. Which is most users. As a result, the income base is much broader and more predictable.
We need better models. And we need to understand our models better.
You're absolutely, completely, 100% right. This is a critical need!
Sad to say, I can think of several models that have been tried - donations, subscriptions, and merchandising come to mind. None require building a subscription apparatus! This makes them morally superior in very important ways, but all of those approaches come with distinct downsides. They all have distortionary effects on the user experience and condition the continued existence on the service on a handful of people.
Perhaps we should stop and think carefully about why surveillence-fuelled ads are so popular an approach. It's a systems that lets one monetize lurkers who are otherwise minimally interactive. Which is most users. As a result, the income base is much broader and more predictable.
We need better models. And we need to understand our models better.
Most users are non-technical so the entire notion is gibberish to them
Otherwise, decentralization imposes performance penalties most deem unacceptable for common use. Furthermore, in the case of p2p, you can end up serving illegal content.
Otherwise, decentralization imposes performance penalties most deem unacceptable for common use. Furthermore, in the case of p2p, you can end up serving illegal content.
This is correct and covers most current attempts at implemention.
The public cannot be confused if you want new technologies to become mainstream. Selling the value of decentralization, something few ordinary people truly care enough to make behavioral or even proactive choices towards adopting, is the wrong strategy currently. That someday will change but it could be a while. The public needs other value propositions like the promise of benefit with little to no effort or mindshare to adopt.
This is similar to why mobile payments faltered (and to some extent still does) in the North American market. Too many players sending out different complex messaging. No unified voice.
Eventually we will eliminate the performance penalties, but mass adoption or demand for a decentralized web will require a complex multi pronged approach that includes education and extensive value adds which are easily understood.
The public cannot be confused if you want new technologies to become mainstream. Selling the value of decentralization, something few ordinary people truly care enough to make behavioral or even proactive choices towards adopting, is the wrong strategy currently. That someday will change but it could be a while. The public needs other value propositions like the promise of benefit with little to no effort or mindshare to adopt.
This is similar to why mobile payments faltered (and to some extent still does) in the North American market. Too many players sending out different complex messaging. No unified voice.
Eventually we will eliminate the performance penalties, but mass adoption or demand for a decentralized web will require a complex multi pronged approach that includes education and extensive value adds which are easily understood.
I like to read the discussion boards for MaidSafe from time to time. [1] In their case at least, serving content would not be mandatory to benefit from the p2p aspect. That's for those running a vault. And even then a vault wouldn't know what it's serving to whom. Nor would the recipient be able to determine the IPs of any of the vaults it connects to.
On the performance side, I think that depends on what's being attempted and how. MaidSafers like to reiterate the notion that popular content will have chunks cached more broadly across the network, but that is only one aspect. There are other things that are not intuitive about how to develop a distributed application with no central authority. And their current model does not include sharing computing power (no Function-as-a-Service type of thing); just storage and bandwidth.
[1] https://safenetforum.org/
On the performance side, I think that depends on what's being attempted and how. MaidSafers like to reiterate the notion that popular content will have chunks cached more broadly across the network, but that is only one aspect. There are other things that are not intuitive about how to develop a distributed application with no central authority. And their current model does not include sharing computing power (no Function-as-a-Service type of thing); just storage and bandwidth.
[1] https://safenetforum.org/
Also decentralisation make fighting abuses hard, such as spam, flood, dos, etc.
On the contrary, this is one of the few areas where decentralized systems are much better than centralized ones.
The problem is that moderation doesn't scale. Small communities and small instances will nearly always be better at moderation than a large platform, because they can rely on a small number of individuals to get the job done - individuals who are more likely to be consistent, more likely to pay attention to community reactions to posts, and more likely to care in general about what they're doing.
It's difficult for me to think of any large platform (Facebook, Github, Twitter, Reddit, Steam, Amazon, etc...) where moderation and general content quality is better than their smaller counterparts (Mastadon, Itch.io, Gitlab, self-hosted forums, etc...).
Large platforms also become large attack vectors. The same benefits of centralization for users - (content discovery, single sources of truth, and so on) make it very cost-effective and efficient to spam and harass users. With a decentralized system, spammers are less likely to care about whatever small community you're hosting. It's not impossible for them to crawl around the internet spamming everyone, but the cost of doing so is a lot higher than targeting a single platform.
So centralization inevitably leads to large platforms because the market for these platforms is winner-take-all, and it almost a kind of pseudo physical law that large platforms can not be moderated well. I don't want to be absolutist about it, but I'm really having a hard time thinking of even a single exception to that rule.
Maybe Wikipedia, but I'm even kind of doubtful about that - and most of Wikipedia's quality moderation comes from a group of people who are completely obsessed with the project. That mindset also doesn't scale to Twitter/Facebook sizes.
The problem is that moderation doesn't scale. Small communities and small instances will nearly always be better at moderation than a large platform, because they can rely on a small number of individuals to get the job done - individuals who are more likely to be consistent, more likely to pay attention to community reactions to posts, and more likely to care in general about what they're doing.
It's difficult for me to think of any large platform (Facebook, Github, Twitter, Reddit, Steam, Amazon, etc...) where moderation and general content quality is better than their smaller counterparts (Mastadon, Itch.io, Gitlab, self-hosted forums, etc...).
Large platforms also become large attack vectors. The same benefits of centralization for users - (content discovery, single sources of truth, and so on) make it very cost-effective and efficient to spam and harass users. With a decentralized system, spammers are less likely to care about whatever small community you're hosting. It's not impossible for them to crawl around the internet spamming everyone, but the cost of doing so is a lot higher than targeting a single platform.
So centralization inevitably leads to large platforms because the market for these platforms is winner-take-all, and it almost a kind of pseudo physical law that large platforms can not be moderated well. I don't want to be absolutist about it, but I'm really having a hard time thinking of even a single exception to that rule.
Maybe Wikipedia, but I'm even kind of doubtful about that - and most of Wikipedia's quality moderation comes from a group of people who are completely obsessed with the project. That mindset also doesn't scale to Twitter/Facebook sizes.
You're right! Gitlab is much better moderated than Facebook. Small, isolated services can be policed strictly in unscalable ways.
Perhaps there's another comparison to be made? Facebook's moderation, for all its many and myriad faults, is infinitely better than the stuff your average SMTP server sees every day.
Federated decentralized systems are a hell that combines the worst of both centralization and decentralization. It's a great way to offer users the features they want, but at the cost of a set of systems where there's no practical way to stem abuse.
Perhaps there's another comparison to be made? Facebook's moderation, for all its many and myriad faults, is infinitely better than the stuff your average SMTP server sees every day.
Federated decentralized systems are a hell that combines the worst of both centralization and decentralization. It's a great way to offer users the features they want, but at the cost of a set of systems where there's no practical way to stem abuse.
I would still disagree with this - federated decentralized services (think Mastadon, or PeerTube) are made up of smaller nodes. Each of those nodes are more tightly policed by smaller communities, and you subscribe to a subset of those communities that you care about.
This is great for moderation - Mastadon instances tend to be very focused and to ban bad actors quickly. It's bad for content discovery, because not all of the content is localized in one place.
It's a tradeoff, one that I'm increasingly suspecting is inherent to the debate over centralization, and one that I think might end up being a deciding factor for people choosing which direction they support. If you care about mass indexing and content discovery, maybe you just can't have moderation. If you care about moderation, maybe you just can't have content discovery or mass indexing.
Granted, decentralization is worse for censorship because bad actors can go off and make their own instance, and other people can still subscribe to them. But, censorship != moderation, those are two entirely different concepts that just occasionally overlap with each other.
This is assuming that you're talking about moderation that's more in-depth than just adding a captcha to defeat spammers. With the example you gave of hosting your own fileshare; Facebook has more resources to deal with a certain kind of attack - DOS, XSS, etc... but you're really only talking about a very small subset of attack patterns in that instance. I think federation makes most of those problems go away, because you can standardize purely technical defenses and for the most part they scale just fine across decentralized services.
The moment you start talking about hard problems of moderation like choosing which files you're going to host in the first place, the private SMTP server wins.
When I think about scaling moderation and curbing abuses, I'm not really thinking about the low hanging fruit, I'm thinking about curbing bad actors who are already actively participating on the platform, or who are coming up with novel attacks that don't have any one-size-fits-all solutions.
This is great for moderation - Mastadon instances tend to be very focused and to ban bad actors quickly. It's bad for content discovery, because not all of the content is localized in one place.
It's a tradeoff, one that I'm increasingly suspecting is inherent to the debate over centralization, and one that I think might end up being a deciding factor for people choosing which direction they support. If you care about mass indexing and content discovery, maybe you just can't have moderation. If you care about moderation, maybe you just can't have content discovery or mass indexing.
Granted, decentralization is worse for censorship because bad actors can go off and make their own instance, and other people can still subscribe to them. But, censorship != moderation, those are two entirely different concepts that just occasionally overlap with each other.
This is assuming that you're talking about moderation that's more in-depth than just adding a captcha to defeat spammers. With the example you gave of hosting your own fileshare; Facebook has more resources to deal with a certain kind of attack - DOS, XSS, etc... but you're really only talking about a very small subset of attack patterns in that instance. I think federation makes most of those problems go away, because you can standardize purely technical defenses and for the most part they scale just fine across decentralized services.
The moment you start talking about hard problems of moderation like choosing which files you're going to host in the first place, the private SMTP server wins.
When I think about scaling moderation and curbing abuses, I'm not really thinking about the low hanging fruit, I'm thinking about curbing bad actors who are already actively participating on the platform, or who are coming up with novel attacks that don't have any one-size-fits-all solutions.
This is the correct answer.
I think that what isn't immediately obvious when most people consider the big silo websites is:
1) The army of moderators that are constantly fighting to remove illegal, offensive, or commercially damaging content. (Twitter might not care about bots posting lies to sway voters, but it doesn't want a reputation that kids are following fake celebrity accounts that post beheading videos every few months).
2) The fact that these sites, by acting as middle men (and mass surveillance infrastructures), get to calculate people's reputation, which is difficult to export between sites in a meaningful way, and is hard to replicate in a decentralised/federated architecture.
I think that what isn't immediately obvious when most people consider the big silo websites is:
1) The army of moderators that are constantly fighting to remove illegal, offensive, or commercially damaging content. (Twitter might not care about bots posting lies to sway voters, but it doesn't want a reputation that kids are following fake celebrity accounts that post beheading videos every few months).
2) The fact that these sites, by acting as middle men (and mass surveillance infrastructures), get to calculate people's reputation, which is difficult to export between sites in a meaningful way, and is hard to replicate in a decentralised/federated architecture.
While I'm not a fan of Steemit, and it has a non-trivial list of flaws imho, I think they represent an interesting counter example to your claim.
The various communities on Steemit all seem to focus rather passionately on protecting their hashtags and feeds from users or content that are abusive or spammy.
The various communities on Steemit all seem to focus rather passionately on protecting their hashtags and feeds from users or content that are abusive or spammy.
> decentralization imposes performance penalties most deem unacceptable for common use...
May impose such penalties; can also provide improvements (closer to point-of-use data, greater resiliency etc).
May impose such penalties; can also provide improvements (closer to point-of-use data, greater resiliency etc).
Actually, the web is decentralized. Many people mess around with the terms here. Just today I wrote a short post defining decentralized, distributed and federated in another thread [1].
While I am completely pro decentralization (I run my own Nextcloud, XMPP and E-Mail server), I am not very fond of all those 'distributed' ideas. Many of them rely on encryption which might be safe to use today, but encryption has this anoying property becoming unsafe in the future. So puting my data encrypted to some distributed network doesn't feel right for me.
The problem with the tech giants is that their products are so easy to start with that many consumers just use them because they don't see real alternatives. And quite frankly, often there are no equivalent alternatives, as the tech giants throw so much money at their products that very few other products have the same ease of use.
I am not sure about a solution to this dilemma, but I hope that some day we will have more high quality, easy to use, federated services.
[1] https://news.ycombinator.com/item?id=17695808
While I am completely pro decentralization (I run my own Nextcloud, XMPP and E-Mail server), I am not very fond of all those 'distributed' ideas. Many of them rely on encryption which might be safe to use today, but encryption has this anoying property becoming unsafe in the future. So puting my data encrypted to some distributed network doesn't feel right for me.
The problem with the tech giants is that their products are so easy to start with that many consumers just use them because they don't see real alternatives. And quite frankly, often there are no equivalent alternatives, as the tech giants throw so much money at their products that very few other products have the same ease of use.
I am not sure about a solution to this dilemma, but I hope that some day we will have more high quality, easy to use, federated services.
[1] https://news.ycombinator.com/item?id=17695808
There are different intepretations or definitions of what is decentralized. For example, Vitalik defines the concwpts in a different way https://medium.com/@VitalikButerin/the-meaning-of-decentrali...
I've been working on decentralised systems of one form or another for a long time. I think there are several ways forward.
One way is to build high quality platforms. Part of why centralised apps tend to be more common than decentralised is that we have pretty good or at least well understood platforms for making centralised apps, but the platforms for making decentralised apps are obscure at best and largely non-existent.
Another way is to narrow and repoint the focus a bit. People thinking about decentralisation tend to immediately gravitate to consumer applications. I've found it easier to get traction in the business-to-business space. Despite all the noise generated by the press consumers often don't really care about things like lockin or data privacy - the tech firms protect people's data well enough, and there isn't usually enough lockin to really worry outside of maybe smartphone operating systems. However businesses are often much more sensitive to these issues. Yes they will outsource and use centralised services too, but they're much more likely to care about things like confidentiality, privacy of their data/trade secrets, they sometimes have regulatory constraints that push them in the direction of decentralisation etc.
So that's one reason why I'm working on Corda, which is a platform for p2p business. It tends to get described as a blockchain system and it's got a lot of tech inspired by Bitcoin but I see it more as a platform for building decentralised apps (or more accurately, apps where you can precisely distribute and control trust points in accordance with your needs).
Better platforms are absolutely the way to go for consumer apps as well. Unfortunately building such platforms will probably involve sacrificing some sacred cows. For instance the web is a terribly centralising platform. It's great at decentralising information and documents but when (ab)used as a way to build ordinary applications it creates far more centralisation than we've ever seen before. To get decentralised apps becoming commonplace we probably need to tackle the dominance of HTML5 and HTTP as a (desktop/laptop) app platform.
One way is to build high quality platforms. Part of why centralised apps tend to be more common than decentralised is that we have pretty good or at least well understood platforms for making centralised apps, but the platforms for making decentralised apps are obscure at best and largely non-existent.
Another way is to narrow and repoint the focus a bit. People thinking about decentralisation tend to immediately gravitate to consumer applications. I've found it easier to get traction in the business-to-business space. Despite all the noise generated by the press consumers often don't really care about things like lockin or data privacy - the tech firms protect people's data well enough, and there isn't usually enough lockin to really worry outside of maybe smartphone operating systems. However businesses are often much more sensitive to these issues. Yes they will outsource and use centralised services too, but they're much more likely to care about things like confidentiality, privacy of their data/trade secrets, they sometimes have regulatory constraints that push them in the direction of decentralisation etc.
So that's one reason why I'm working on Corda, which is a platform for p2p business. It tends to get described as a blockchain system and it's got a lot of tech inspired by Bitcoin but I see it more as a platform for building decentralised apps (or more accurately, apps where you can precisely distribute and control trust points in accordance with your needs).
Better platforms are absolutely the way to go for consumer apps as well. Unfortunately building such platforms will probably involve sacrificing some sacred cows. For instance the web is a terribly centralising platform. It's great at decentralising information and documents but when (ab)used as a way to build ordinary applications it creates far more centralisation than we've ever seen before. To get decentralised apps becoming commonplace we probably need to tackle the dominance of HTML5 and HTTP as a (desktop/laptop) app platform.
tech giants throw so much money at their products that very few other products have the same ease of use
If you have a typical-sized online service, I think you can be in the same league as the tech giants with two full-time employees focused on ease of use. With four I think you can be on equal footing. And if you charge users for your service, you can focus on ease-of-use improvements that let users get things done faster. A tech giant whose revenue increases with "engagement" is going to make few improvements that help people leave their site faster.
EDIT: To be clear, I'm talking about competing on ease of use, not feature count.
If you have a typical-sized online service, I think you can be in the same league as the tech giants with two full-time employees focused on ease of use. With four I think you can be on equal footing. And if you charge users for your service, you can focus on ease-of-use improvements that let users get things done faster. A tech giant whose revenue increases with "engagement" is going to make few improvements that help people leave their site faster.
EDIT: To be clear, I'm talking about competing on ease of use, not feature count.
It's already here. I'm not sure why Danielle Robinson merited a photo and a quote but there was no mention of Beaker Browser. Beaker uses Dat as its storage layer, which is similar to IPFS or Bittorrent, and then layers on data storage and sharing systems to enable services like Fritter, their Twitter clone. It's very easy to use -- just download Beaker, and you can be publishing web pages or using totally decentralized web applications.
+1 - where IPFS and Gun present a very tech-focused entry point to the distributed web (I still do not even know how Gun works - it just says it works and that I dont need to worry etc, without much info that I could find for actually how it works e.g. how is my data shared? who runs the other nodes), Beaker presents a very friendly "on ramp" to it all that is easily understandable in a browser context. I've found it very interesting to play around with. Highly recommended.
I spoke at the conference this week, it was mind blowing to meet all these people. :)
The article seemed well done, the problem (from a information theory / network graph) though seemed weighted towards blockchain-like assumptions. This is fine, as more information needs to be seeded on this.
For those interested in the title's question:
Coordination might seem like the hard problem, but it is only hard because the majority of us in this community made it hard by using immutable data structures. If we use mutable data structures (you know my bias, but it is possible with https://github.com/amark/gun , proof: Mitra at Internet Archive integrated it in 1 week, now decentralized IA runs on GUN), we get some other problems but they are less than O(log N) in complexity.
Here is why coordination on immutability is hard:
O(N^2)
Just sit and think about it.
Intuitively it makes sense:
If I have an index to make something fast, and if it is immutable.
When I "update" the index, I have to create a new index.
Now the old index can't find the new index.
Therefore, I need to index the index, etc.
Repeat.
Therefore, as long as the web never changes, a decentralized immutable web will eventually be fast, but it must be true that if while the decentralized web changes, it will be hard.
This is not true if you can do decentralized mutable data. Check out our work! I hope this comment added novel insight to your day. :) If it did, let me buy you coffee next time you are in town. Cheers!
The article seemed well done, the problem (from a information theory / network graph) though seemed weighted towards blockchain-like assumptions. This is fine, as more information needs to be seeded on this.
For those interested in the title's question:
Coordination might seem like the hard problem, but it is only hard because the majority of us in this community made it hard by using immutable data structures. If we use mutable data structures (you know my bias, but it is possible with https://github.com/amark/gun , proof: Mitra at Internet Archive integrated it in 1 week, now decentralized IA runs on GUN), we get some other problems but they are less than O(log N) in complexity.
Here is why coordination on immutability is hard:
O(N^2)
Just sit and think about it.
Intuitively it makes sense:
If I have an index to make something fast, and if it is immutable.
When I "update" the index, I have to create a new index.
Now the old index can't find the new index.
Therefore, I need to index the index, etc.
Repeat.
Therefore, as long as the web never changes, a decentralized immutable web will eventually be fast, but it must be true that if while the decentralized web changes, it will be hard.
This is not true if you can do decentralized mutable data. Check out our work! I hope this comment added novel insight to your day. :) If it did, let me buy you coffee next time you are in town. Cheers!
You don't have to copy the whole datastructure to update it, even if it's immutable.
In something like an append-only log or DAG, you'd still have to index where in that log to skip to (or you'd be scanning for it every time), which is still computational complex, and not nearly as scalable as a mutable index. So yes, while you are correct, it either way is still better to use the mutable approach.
Napster, Limeware, Bittorrent, Kodi, Bitcoin, were/are part of the 'decentralized web'. They were build by necessity to get around copyright, monetary restrictions, censorship etc.. and not necessary for convenience.
At the end of the day the average user just wants a convenient and reliable way to reach content and information. Centralization has been the best way to serve it, and I don't think it will change.
At the end of the day the average user just wants a convenient and reliable way to reach content and information. Centralization has been the best way to serve it, and I don't think it will change.
Napster was by far the best way to share music, just like how popcorn time is the easiest way to get a movie right here right now. However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient (while it isn't). Unfortunately we 're not gonna see decentralized web unless the general public gets back control of the compute and traffic.
Here in Germany at least the frequent expensive letters you get when you run a file sharing node are probably a more important reason for the demise of file sharing than the rise of smartphones.
>However nowadays people ditched their desktop computer/node for an iphone, and ISP upload speeds are in most places pretty bad. I think this has crippled p2p services and made centralization more convenient
Most people were on dialup in Napster's heyday, so I don't think that can be the issue. Music files are small.
Most people were on dialup in Napster's heyday, so I don't think that can be the issue. Music files are small.
but phones do not act as always-on nodes like napster or kazaa or torrent nodes did. battery issues + mobile data limits also
This is probably why it's not only the Daughters Bell and Comcast encouraging FCC to stomp the brakes on modern radio tech. MPAA et al. also like to keep network consumers in a weak position. Really though it's silly for us to expect this sort of ecosystem to thrive on phones. It would more naturally thrive on the doodads that plug into TVs. That's where Kodi mainly is now, isn't it?
I feel like decentralization is once again a necessity because companies like Facebook can't be trusted.
I feel most people would jump ship if there was a viable, easy to use, alternative where you had ownership of your own data.
I feel most people would jump ship if there was a viable, easy to use, alternative where you had ownership of your own data.
Most people use products, not protocols. We need a more product-oriented approach. Decentralization and federation aren't the first or second item on the list of things most people are looking for, unless we get to "exactly like X, but decentralized."
It's wrong that people don't understand or want to not depend on something like Facebook. At least where I am, a lot of people understand the issues. But there's nothing really like it, and life in my city is quite difficult without.
It's nothing that can't be fixed though! I think a project like Pixelfed (https://pixelfed.org/) is doing things right. We can get there.
It's wrong that people don't understand or want to not depend on something like Facebook. At least where I am, a lot of people understand the issues. But there's nothing really like it, and life in my city is quite difficult without.
It's nothing that can't be fixed though! I think a project like Pixelfed (https://pixelfed.org/) is doing things right. We can get there.
> Most people use products, not protocols.
Except e-mail and HTTP.
> We need a more product-oriented approach.
I think this is how we landed in the current situation. Instead of protocols (IRC, XMPP), you get products (Slack, Hipchat), which are walled gardens. I wish there was a way to incentivize companies to stick to shared protocols, instead of creating their own.
Except e-mail and HTTP.
> We need a more product-oriented approach.
I think this is how we landed in the current situation. Instead of protocols (IRC, XMPP), you get products (Slack, Hipchat), which are walled gardens. I wish there was a way to incentivize companies to stick to shared protocols, instead of creating their own.
The people use products, which then implement some protocol. If you go down to the nearest supermarket and ask what people use to visit their news website of choice, I'm willing to bet that 99+% will name a browser or a phone type and a very very small minority will answer HTTP, even though technically all of them use HTTP. You'll get similar answers with email and SMTP.
The general public is just not very interested in the implementation details of technical products, as long as they work well enough and don't cost a lot. If you really want incentivize companies to implement open protocols, find a way for companies to make more money with the open protocol than with the walled garden, because companies is where most of the engineers are and they are the ones that will make it if the incentive is there.
The general public is just not very interested in the implementation details of technical products, as long as they work well enough and don't cost a lot. If you really want incentivize companies to implement open protocols, find a way for companies to make more money with the open protocol than with the walled garden, because companies is where most of the engineers are and they are the ones that will make it if the incentive is there.
Moxie Marlinspike has written about this. It's hard to get people to use shared protocols because they're so hard to iterate. Companies that have a single frontend and single backend and a private protocol they control, can add new features far faster than can be done in a centralised approach.
These days I'm more interested in solutions where companies produce decentralised systems, and they provide the leadership and drive necessary, with the right safeguards in place to allow their leadership to be removed and replaced if necessary. Rather than the more traditional approach of trying to not have any leadership, or do it all by committee.
These days I'm more interested in solutions where companies produce decentralised systems, and they provide the leadership and drive necessary, with the right safeguards in place to allow their leadership to be removed and replaced if necessary. Rather than the more traditional approach of trying to not have any leadership, or do it all by committee.
In the old days, you were a member of an organisation such as a university or a corporation. You had a home directory on an NFS server and in your home dir was a folder called public_html that you put your homepage in. So your URL was www.university.ac.uk/~username and your email address was [email protected]
This caused problems whenever anyone moved so in the early days centralised services were seen as a solution such as Hotmail and Geocities. No one in 1995 realised how it would turn out into abusive monopolies...
This caused problems whenever anyone moved so in the early days centralised services were seen as a solution such as Hotmail and Geocities. No one in 1995 realised how it would turn out into abusive monopolies...
I think a decentralized web is easier than what people think. In the early days of the web, users could host their creations somewhere in their ISP's hard disks.
The same model could work now, with a small difference: instead of storing only data, it may also host applications that manage those data, and collaborate with same/other applications of same/other users.
For example, a photo app could allow us to manage our photos at a specific host of our choice; the same photo app uses a sharing protocol so as that the photos are shared with other users of our choice.
The stored data are encrypted by our own keys and therefore not visible to third parties, we still get to share our stuff with our mates, it's decentralized because no central authority can lay claim on our data, it's protected because it is encrypted for us, but the infrastructure required for this to run is provided by a centralized entity.
The same model could work now, with a small difference: instead of storing only data, it may also host applications that manage those data, and collaborate with same/other applications of same/other users.
For example, a photo app could allow us to manage our photos at a specific host of our choice; the same photo app uses a sharing protocol so as that the photos are shared with other users of our choice.
The stored data are encrypted by our own keys and therefore not visible to third parties, we still get to share our stuff with our mates, it's decentralized because no central authority can lay claim on our data, it's protected because it is encrypted for us, but the infrastructure required for this to run is provided by a centralized entity.
The UI/UX usually sucks a lot.
Bitcoin wallets for example, suffer a lot because the average user doesn't have much of an idea that they need a strong password or what these random numbers mean other that they might behave like bank accounts.
The ideal decentralized UI/UX will work largely by copy paste, with human readable names (@[email protected], for example) and doesn't have too many buttons that need more than an icon or single word to explain them.
On top of that it also needs to look modern and neat while being fast and snappy on mobile devices without sucking up battery or limiting how the users can interact (ie, not possible to share photos directly with the app is a no-go)
These design limitations are solved more easily by centralized services as they can just throw money at it.
Bitcoin wallets for example, suffer a lot because the average user doesn't have much of an idea that they need a strong password or what these random numbers mean other that they might behave like bank accounts.
The ideal decentralized UI/UX will work largely by copy paste, with human readable names (@[email protected], for example) and doesn't have too many buttons that need more than an icon or single word to explain them.
On top of that it also needs to look modern and neat while being fast and snappy on mobile devices without sucking up battery or limiting how the users can interact (ie, not possible to share photos directly with the app is a no-go)
These design limitations are solved more easily by centralized services as they can just throw money at it.
'Decentralising' means spreading around effective decision-making power. That power comes with consequences -- negatives ones, if ineptly used. And most users are quite inept, when it comes to running powerful general-purpose computers, especially internet-connected ones. I don't blame them: the internet is not a friendly environment. It's a jungle. People want to be able to safely communicate online without the responsibility of system administration. Many companies have stepped in to offer services with this trade-off. Thus centralisation has prevailed.
Urbit's conceptual path to decentralisation seems sound: build a new friendly network on top of the internet, and build a new friendly server OS on top of Unix.
Urbit's conceptual path to decentralisation seems sound: build a new friendly network on top of the internet, and build a new friendly server OS on top of Unix.
It's not hard, but many in the space are working on the wrong things. We need a consensus mechanism with the integration capability of tendermint, with honey badger bft level performance. (eg: https://github.com/poanetwork/hbbft) That would be enough to build a fully distributed facebook clone and much more. It will happen, but on the timing of the engineers who know what to do to achieve it.
The problem here is decentralized network relies on peers to self-enforce their decent behavior.
End users don't know how to, and their vendors have no skin in the game - then we get IoT botnets.
ISPs won't bother with egress filtering - then we get massive DoS attacks.
Centralization is comfortable and in the end run, cheaper. I'd rather pay github (or depending on how much Microsoft fucks it up, a competitor in the future) a couple dollars per month than maintain my own git server.
Even for services where I have to run my own server, I rent a VPS at some hosting facility rather than maintaining my own hardware or even renting bare metal. It simply doesn't make sense to run my own.
And if I bothered to have a web presence, again, I'd host it somewhere. It simply doesn't make sense to maintain your own except as a hobby.
Moreover, all the blockchain stuff looks to me like a solution looking for a problem instead of something generally useful. Yes, it will make sense for decentralized payments if the exchange rates ever stabilize. No, adding blockchain to everything won't help.
Even for services where I have to run my own server, I rent a VPS at some hosting facility rather than maintaining my own hardware or even renting bare metal. It simply doesn't make sense to run my own.
And if I bothered to have a web presence, again, I'd host it somewhere. It simply doesn't make sense to maintain your own except as a hobby.
Moreover, all the blockchain stuff looks to me like a solution looking for a problem instead of something generally useful. Yes, it will make sense for decentralized payments if the exchange rates ever stabilize. No, adding blockchain to everything won't help.
I think key to this distinction is federation, not decentralisation. With open standards and open source implementations, you'd be able to migrate your GitHub to a different GitHub provider if Microsoft screws things up too badly. GitHub would be not a monosilo, but a set of slightly smaller siloes, all interacting seamlessly with each other.
Well, I use github just for the repo part, no other fancy features. We run our own issue tracker on a VPS;)
You are mixing up a whole lot of stuff here. The relevant question with regards to centralization is how power is distributed. In that regard, hosting your website with some hosting service is not the same thing as using Github or Facebook.
Hosting your website with a hosting service has no network effect for the hosting company. If they misbehave, you can simply move to a different hosting company with absolutely no change visible to the outside. You keep your domain, you are still reachable using the same web browsers at the same URI, via the same bookmarks, via the same hyperlinks.
So, while that arguably is more centralized than running your own hardware at home, it's nowhere near the level of centralization of power with services that have a business model of building a monopoly, such as Github or Facebook. If Facebook mistreats you, you cannot leave Facebook and still keep using the service they are offering.
Especially with the example of git hosting, the primary problem isn't the hosting part, it's the specific business model of Github. The problem isn't that you are delegating the hosting to a specialized business. The problem is that you are delegating it to a business that's holding your hostage and using you as leverage against the general public.
Also, you are making some assumptions that might be worth questioning. Like, why doesn't it make sense to maintain your own web server? I assume you assume that takes a lot of tinkering that you don't want to bother with. But why would that be the case? Is that a technical necessity? Or is that just some experience of yours that doesn't really have to be that way? Is there any reason why hosting your own server couldn't be more reliable and less work than using a hosting service?
Hosting your website with a hosting service has no network effect for the hosting company. If they misbehave, you can simply move to a different hosting company with absolutely no change visible to the outside. You keep your domain, you are still reachable using the same web browsers at the same URI, via the same bookmarks, via the same hyperlinks.
So, while that arguably is more centralized than running your own hardware at home, it's nowhere near the level of centralization of power with services that have a business model of building a monopoly, such as Github or Facebook. If Facebook mistreats you, you cannot leave Facebook and still keep using the service they are offering.
Especially with the example of git hosting, the primary problem isn't the hosting part, it's the specific business model of Github. The problem isn't that you are delegating the hosting to a specialized business. The problem is that you are delegating it to a business that's holding your hostage and using you as leverage against the general public.
Also, you are making some assumptions that might be worth questioning. Like, why doesn't it make sense to maintain your own web server? I assume you assume that takes a lot of tinkering that you don't want to bother with. But why would that be the case? Is that a technical necessity? Or is that just some experience of yours that doesn't really have to be that way? Is there any reason why hosting your own server couldn't be more reliable and less work than using a hosting service?
I don't see decentralization as ultimate goal. Why should it be?
Because humanity's experience with centralization of power has been abysmal--and unsurprisingly so.
Why do the articles on the site require sign up to read? prime example of friction stopping the web from being decentralized.
Decentralisation is hard because people crave power. As long as people crave power, people will build centralised systems of control.
Also, because people crave money. It is more difficult to extract money from decentralized systems compared to centralized systems. Therefore centralized solutions often have more resources available to pay developers, marketeers and founders (and investors), which leads to increased work on the product and (hopefully) more users. It is easy to see how this would lead to a virtuous cycle where the centralized systems capture the majority of the market due to having vastly superior resources and the decentralized systems capture mostly those users and developers who care about things other than money (eg censorship resistance, etc).
Money is power. Money is quite literally the ability to make other people work for the things you want.
Money is a form of power, sure. There are different forms of power as well, not all of which can (realistically) be bought. Try purchasing serious weaponry for example (let's say a dozen tanks or fighter jets) and you will quickly discover that money is not enough.
Centralization is so useful that people will create it wherever it does not already exist.
One example is Github, a centralized service that makes it easier for teams to use a source control system that was originally designed to be decentralized and distributed. When it was purchased by Microsoft and people started looking elsewhere, they looked to competing centralized solutions like GitLab and Bitbucket.
Given that centralization is so useful, and people want it so much, it should not be resisted. We would be better off if the effort being put into decentralization was instead put into better centralized services that respect user privacy.
One example is Github, a centralized service that makes it easier for teams to use a source control system that was originally designed to be decentralized and distributed. When it was purchased by Microsoft and people started looking elsewhere, they looked to competing centralized solutions like GitLab and Bitbucket.
Given that centralization is so useful, and people want it so much, it should not be resisted. We would be better off if the effort being put into decentralization was instead put into better centralized services that respect user privacy.
Monopolies are so useful that markets will create them wherever they do not already exist.
One example is Microsoft, a monopolistic company that makes an operating system that everyone uses, replacing lots of unpopular incompatible ones.
Given that monopolies are so useful, and people want them so much, they should not be resisted. We would be better off if the effort being put into making markets competitive was instead put into increasing Microsoft's share price.
One example is Microsoft, a monopolistic company that makes an operating system that everyone uses, replacing lots of unpopular incompatible ones.
Given that monopolies are so useful, and people want them so much, they should not be resisted. We would be better off if the effort being put into making markets competitive was instead put into increasing Microsoft's share price.
The problem solved by Github's centralization was establishing a namespace, which could have been done in several other ways. (In fact there are a number of other namespaces orthogonal to Github's but pointing at largely the same resources.) The problem Github solved that convinced lots of OSS people to use it was a reliable offsite that was free as in beer and less obnoxious than Sourceforge. So far it seems like some combination of VC cash and paid usage sustains Github's free service, and several other firms' free services as well. The centralization bit doesn't seem so important.
It's a lot more than just a reliable offsite -- it invented collaboration features that didn't exist in pure Git. Github's pull request and code review workflow has become pervasive in the industry.
It's also not sustained by VC cash anymore -- Microsoft is buying the company.
It's also not sustained by VC cash anymore -- Microsoft is buying the company.
Sure the collaboration features are also valuable. It also isn't obvious that they require centralization. One could argue that only a centralizing organization could extract the rents that could underwrite developing those features, but such an argument only goes so far.
More likely: centralisation has its pros and cons, as does decentralisation. There will always be some forces pushing towards centralisation, and other forces pushing towards decentralisation. The further we move towards one, the greater the push will be towards the other.
What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?