The Shirky Principle (2010)(kk.org)
kk.org
The Shirky Principle (2010)
http://kk.org/thetechnium/the-shirky-prin/
49 comments
>> Unions, or example.
Unions were defanged, waylaid and driven into the ground by neoliberal policies in the US and the UK, in particular. They lost most of their power and were rendered incapable of supporting workers' rights, which was their reason to exist. That is the problem with unions, not that they perpetuate the problems they solve. For that, we have free-market capitalism.
Unions were defanged, waylaid and driven into the ground by neoliberal policies in the US and the UK, in particular. They lost most of their power and were rendered incapable of supporting workers' rights, which was their reason to exist. That is the problem with unions, not that they perpetuate the problems they solve. For that, we have free-market capitalism.
One of the scariest examples of this phenomenon is the lobbying that police and prison guard unions do:
https://theintercept.com/2016/05/18/ca-marijuana-measure/
https://www.nytimes.com/1995/11/07/us/political-gains-by-pri...
https://theintercept.com/2016/05/18/ca-marijuana-measure/
https://www.nytimes.com/1995/11/07/us/political-gains-by-pri...
... until you think about what this means for CPS, and mental health services/institutions. Then the police actions seem positively polite.
The post linked in the post is a better read:
http://www.shirky.com/weblog/2010/04/the-collapse-of-complex...
http://www.shirky.com/weblog/2010/04/the-collapse-of-complex...
[deleted]
I've worked in a lot of different organizations and have seen this happen over and over again.
One point the author misses is that it becomes received knowledge that certain parts of the problem are insolvable. After a dozen or two meetings trying to solve something, the participants believe (and tell one another) that this part of the problem can't be solved.
There's a certain pivot point that happens when you join an organization. At first, you're asking "Why can't we do X?" and "If we do Y, won't that fix this part of the problem?"
Each time, you are told that yes, we tried that. Several times. Really smart people tried. It didn't work.
Now the trick here is that you don't know if that's true or not. You don't know how many times it was tried, who tried it, what strategy they used, or exactly how things played out. It's all just secondhand information.
Do that enough? Pretty soon you're the person telling the new guy that we've already been down this road. It just can't be done. You stop being part of the solution and start being part of the problem. (And then some new organization comes along and solves the problem quite nicely. Cue up a lot of "Well, we could do it to if we only had Z like they did!" sour grapes talk.)
For an outsider trying to help people, when you hear yourself doing this, it's time to go. You are no longer a positive agent of change.
One point the author misses is that it becomes received knowledge that certain parts of the problem are insolvable. After a dozen or two meetings trying to solve something, the participants believe (and tell one another) that this part of the problem can't be solved.
There's a certain pivot point that happens when you join an organization. At first, you're asking "Why can't we do X?" and "If we do Y, won't that fix this part of the problem?"
Each time, you are told that yes, we tried that. Several times. Really smart people tried. It didn't work.
Now the trick here is that you don't know if that's true or not. You don't know how many times it was tried, who tried it, what strategy they used, or exactly how things played out. It's all just secondhand information.
Do that enough? Pretty soon you're the person telling the new guy that we've already been down this road. It just can't be done. You stop being part of the solution and start being part of the problem. (And then some new organization comes along and solves the problem quite nicely. Cue up a lot of "Well, we could do it to if we only had Z like they did!" sour grapes talk.)
For an outsider trying to help people, when you hear yourself doing this, it's time to go. You are no longer a positive agent of change.
This is all true, I’ve seen it many times as well. But a note of caution I would sound for software developers who find themselves going through this cycle in a business, because we technologists are often guilty of not matching our priorities to the business need.
You join a company, and you explore their codebase, and of course you see dozens of obvious opportunities to improve. Abstractions that could unlock new capabilities, systems that don’t communicate that could be integrated, code that could be optimized, obvious user scenarios that were ignored or left unfinished.
Yet the business keeps asking for other things to be done, ignoring these obvious low-hanging opportunities! What fools!
Stop. Take a step back. Look at the numbers. It is often the case that the improvements you can see are worth hundreds of thousands of dollars, but the business is pursuing hundreds of millions. There can be orders of magnitude of difference that, if you’re honest, mean that working on those obvious opportunities is a waste of time compared to the big picture.
It can even be the case that a competitor shows up and seemingly validates one of those opportunities you’re foregoing - you, a frustrated developer, point to this upstart business building something and you say ‘look! We could have built that!’
Then you look closely and realize they’re making pennies in a tiny market and meanwhile your business grew by more than ten times their total revenue in the last year.
Of course, I’m not saying it’s always right to ignore these opportunities; just that when you first join a company, you need to understand not just the technical space but also the business before you start proposing how to divert resources.
You join a company, and you explore their codebase, and of course you see dozens of obvious opportunities to improve. Abstractions that could unlock new capabilities, systems that don’t communicate that could be integrated, code that could be optimized, obvious user scenarios that were ignored or left unfinished.
Yet the business keeps asking for other things to be done, ignoring these obvious low-hanging opportunities! What fools!
Stop. Take a step back. Look at the numbers. It is often the case that the improvements you can see are worth hundreds of thousands of dollars, but the business is pursuing hundreds of millions. There can be orders of magnitude of difference that, if you’re honest, mean that working on those obvious opportunities is a waste of time compared to the big picture.
It can even be the case that a competitor shows up and seemingly validates one of those opportunities you’re foregoing - you, a frustrated developer, point to this upstart business building something and you say ‘look! We could have built that!’
Then you look closely and realize they’re making pennies in a tiny market and meanwhile your business grew by more than ten times their total revenue in the last year.
Of course, I’m not saying it’s always right to ignore these opportunities; just that when you first join a company, you need to understand not just the technical space but also the business before you start proposing how to divert resources.
Programmers interests aren't aligned with the business because they don't directly share in the profits in a significant way. Managers do and so their interests are somewhat aligned with the stockholders. Instead, the programmers do what is in their self-interest, make their own lives better since they see little direct benefit from doing otherwise.
The stronger the incentives to be helpful, the more helpfulness you'll get.
The stronger the incentives to be helpful, the more helpfulness you'll get.
This phenomenon shows how culture and norms perpetuate in societies and companies. And it's notable that it does not originate with senseless malice in the the actors' behavior, but with rational self-preservation.
In company culture, this phenomenon plays out because new entrants would likely hit the same institutional walls, encounter the same defeat, then get disenchanted and leave. The preemptive conditioning is a tool others use in good faith to help a new entrant set realistic expectations, and focus their energy somewhere where it has a higher likelihood of success.
The parallels to the Peters Principle are interesting. It can be controversial to flag phenomenon certain phenomena as self-fulfilling, but there's tautological comfort in declaring everyone's at their current job because they haven't left yet. The Peters Principle explains that by supposing that promotions to management roles tend to be based on performance in non-management roles from which skills don't necessarily transfer, and middling managers no longer get promoted further; this one is about group behavior tends towards further conformity, as disruptors are pacified by others or get frustrated and depart. Over time, the survivors of the group will tend to be more similar than different.
In company culture, this phenomenon plays out because new entrants would likely hit the same institutional walls, encounter the same defeat, then get disenchanted and leave. The preemptive conditioning is a tool others use in good faith to help a new entrant set realistic expectations, and focus their energy somewhere where it has a higher likelihood of success.
The parallels to the Peters Principle are interesting. It can be controversial to flag phenomenon certain phenomena as self-fulfilling, but there's tautological comfort in declaring everyone's at their current job because they haven't left yet. The Peters Principle explains that by supposing that promotions to management roles tend to be based on performance in non-management roles from which skills don't necessarily transfer, and middling managers no longer get promoted further; this one is about group behavior tends towards further conformity, as disruptors are pacified by others or get frustrated and depart. Over time, the survivors of the group will tend to be more similar than different.
Because I've seen a lot of orgs of all sizes, people sometimes ask what happens to stress levels as the org grows? In fiction, big companies are high-risk, high-drama affairs.
Not so much. In reality, very quickly the stress almost completely disappears. Instead of there being a bunch of things to stress over, there become a very few -- they're mostly hidden. As soon as you grow to even 20 or 30 people, members of the group start actively doing all they can to reduce stress. As you already know, the people who tell you that it's never going to work aren't your enemy, and they really want it fixed. They're doing their best to make your job productive and happy.
I worked in a very well-known financial firm once that had just gone through several rounds of lay-offs. It was one of the most encouraging places I've seen. Whatever was suggested, people loved it. They didn't change what they were doing, but they weren't about to try things they "knew" weren't going to work, so they offered effusive praise and found good places to hide.
Big companies may do cruel and inhumane things to their employees, but it's most always out of kindness, not malice. The operative rule is "no drama" -- which can many times mean "don't rock the boat". The two get conflated quite a bit.
Not so much. In reality, very quickly the stress almost completely disappears. Instead of there being a bunch of things to stress over, there become a very few -- they're mostly hidden. As soon as you grow to even 20 or 30 people, members of the group start actively doing all they can to reduce stress. As you already know, the people who tell you that it's never going to work aren't your enemy, and they really want it fixed. They're doing their best to make your job productive and happy.
I worked in a very well-known financial firm once that had just gone through several rounds of lay-offs. It was one of the most encouraging places I've seen. Whatever was suggested, people loved it. They didn't change what they were doing, but they weren't about to try things they "knew" weren't going to work, so they offered effusive praise and found good places to hide.
Big companies may do cruel and inhumane things to their employees, but it's most always out of kindness, not malice. The operative rule is "no drama" -- which can many times mean "don't rock the boat". The two get conflated quite a bit.
Isn't there a famous monkey experiment that describes this situation. Long after the original monkey that exhibits some (natural monkey) action and gets reprimanded for it, other monkeys try to stop new monkeys coming into the group from doing the same action generations later, even after the original monkey is no longer in the group and the action is no longer reprimanded against.
Your link indicates that there is no evidence that this experiment ever took place.
[deleted]
Similar to this experiment, where people continue to follow an order even after the original group members have left: https://www.youtube.com/watch?v=keBh_LK5Ha4
I have been on both sides of this. And both sides are often right.
In defense of defenders of status quo, after some time, it becomes tedious to explain the reasons why it didn't work, to people who didn't do basic homework. Sometimes the turnaround of people is just too high. And the defenders of status quo often have their own ideas of what are important problems to solve, and they are often right.
On the other hand, the innovator shouldn't just quit. If you think your idea has merit, build a prototype! Prototype of almost anything can be built by a single person. Maybe it can be done after all.
I think the solution to this problem is 20% time to work on whatever (improvement) you want. The youngsters can then work on new innovative things, which the older guys considered impossible. And the older guys can work on what they really think matters.
In defense of defenders of status quo, after some time, it becomes tedious to explain the reasons why it didn't work, to people who didn't do basic homework. Sometimes the turnaround of people is just too high. And the defenders of status quo often have their own ideas of what are important problems to solve, and they are often right.
On the other hand, the innovator shouldn't just quit. If you think your idea has merit, build a prototype! Prototype of almost anything can be built by a single person. Maybe it can be done after all.
I think the solution to this problem is 20% time to work on whatever (improvement) you want. The youngsters can then work on new innovative things, which the older guys considered impossible. And the older guys can work on what they really think matters.
I think the saying “Institutions will try to preserve the problem to which they are the solution” is for institutions not for individuals. The idea is that for instance The Society to End Hunger will never end hunger because if it does it will have to end its existence. I think what you are saying is looking for another good aphorism. I'm sure there must be one.
Hello pharma industry
Yes but ...
Big pharma currently doesn't want to cure medical conditions, it wants to sell drugs continuously. (See also: lack of development of new antibiotics, huge over-investment in anti-depressants, etc.)
New medicines are very expensive to develop because of the extreme regulatory environment and relatively short patent duration (two decades from registering a patent on a drug to it going into a free-for-all of cheap generics: development takes 8-16 years and the remaining 4-12 years are all the time the developer has got to amortize their costs and turn a profit).
But the regulations and requirements for extensive testing exist for very good reason, to prevent this sort of thing from happening:
https://en.wikipedia.org/wiki/Elixir_Sulfanilamide (1938: at least 100 dead)
https://en.wikipedia.org/wiki/Thalidomide#History (1950s: use for morning sickness caused >2000 deaths, 10,000 born with severe deformities)
https://en.wikipedia.org/wiki/Vioxx (2004: 88-140,000 cases of serious heart disease)
There's no solution to the problem of complex, emergent, lethal drug side-effects on the horizon. So we have to have expensive clinical trials and statistical tests to rule out a high risk of harm before a new drug can be released. So the pharmaceutical industry has a baked-in incentive to maximize revenue by ignoring certain conditions (brief/acute illnesses, especially those afflicting the poor or the developing world) and prioritize chronic diseases of affluence. There's no fix within the established framework, because the established framework is defective by emergent design.
We could tinker with the international intellectual property regime governing pharmaceutical patent duration, but that kicks the ball up to WTO treaty level. Or we could consider non-profit bases for pharmaceutical R&D, but that goes against the current dominant free market ideology.
Big pharma currently doesn't want to cure medical conditions, it wants to sell drugs continuously. (See also: lack of development of new antibiotics, huge over-investment in anti-depressants, etc.)
New medicines are very expensive to develop because of the extreme regulatory environment and relatively short patent duration (two decades from registering a patent on a drug to it going into a free-for-all of cheap generics: development takes 8-16 years and the remaining 4-12 years are all the time the developer has got to amortize their costs and turn a profit).
But the regulations and requirements for extensive testing exist for very good reason, to prevent this sort of thing from happening:
https://en.wikipedia.org/wiki/Elixir_Sulfanilamide (1938: at least 100 dead)
https://en.wikipedia.org/wiki/Thalidomide#History (1950s: use for morning sickness caused >2000 deaths, 10,000 born with severe deformities)
https://en.wikipedia.org/wiki/Vioxx (2004: 88-140,000 cases of serious heart disease)
There's no solution to the problem of complex, emergent, lethal drug side-effects on the horizon. So we have to have expensive clinical trials and statistical tests to rule out a high risk of harm before a new drug can be released. So the pharmaceutical industry has a baked-in incentive to maximize revenue by ignoring certain conditions (brief/acute illnesses, especially those afflicting the poor or the developing world) and prioritize chronic diseases of affluence. There's no fix within the established framework, because the established framework is defective by emergent design.
We could tinker with the international intellectual property regime governing pharmaceutical patent duration, but that kicks the ball up to WTO treaty level. Or we could consider non-profit bases for pharmaceutical R&D, but that goes against the current dominant free market ideology.
> Big pharma currently doesn't want to cure medical conditions, it wants to sell drugs continuously. (See also: lack of development of new antibiotics, huge over-investment in anti-depressants, etc.)
I'm so tied of hearing this claim.
Gilead created a cure for hep C and tried to charge users X% of the lifetime cost of the existing interfeuron therapy which only managed symptoms but was not a cure. You would expect that this would be received favorably: patient gets cured, insurance companies save money over the long term, Gilead makes a healthy profit. Wrong; people flipped a shit, screaming about how this was price gouging and that a single treatment should never be this expensive. Gilead has been embattled ever since. If you want pharma companies to make cures, inventivize them to do so.
I'm so tied of hearing this claim.
Gilead created a cure for hep C and tried to charge users X% of the lifetime cost of the existing interfeuron therapy which only managed symptoms but was not a cure. You would expect that this would be received favorably: patient gets cured, insurance companies save money over the long term, Gilead makes a healthy profit. Wrong; people flipped a shit, screaming about how this was price gouging and that a single treatment should never be this expensive. Gilead has been embattled ever since. If you want pharma companies to make cures, inventivize them to do so.
>> If you want pharma companies to make cures, inventivize them to do so.
You stopped just short of a breakthrough. The institution of for-profit healthcare is something we could do without. No one should suffer or die because they couldn't get rich enough before facing a life-limiting illness or injury.
You stopped just short of a breakthrough. The institution of for-profit healthcare is something we could do without. No one should suffer or die because they couldn't get rich enough before facing a life-limiting illness or injury.
sparkzilla(2)
[deleted]
X% of lifetime cost isn't meaningful without stating the lifetime cost.
What is the lifetime cost?
What is the lifetime cost?
+1 for coining the word inventivize. I can't wait to use this for something, even tho I don't yet know what it means
> Or we could consider non-profit bases for pharmaceutical R&D, but that goes against the current dominant free market ideology.
Considering the huge costs of healthcare in western countries why don't all the states or maybe EU cooperate? This seems to happen in research but why stop there? Add 1% of tax for it - it's probably worth it - alone on the fact you have an organisation without profit motivation.
However even if realized I guess lobbying will destroy any good that might come out of such an initiative before it's even started.
Considering the huge costs of healthcare in western countries why don't all the states or maybe EU cooperate? This seems to happen in research but why stop there? Add 1% of tax for it - it's probably worth it - alone on the fact you have an organisation without profit motivation.
However even if realized I guess lobbying will destroy any good that might come out of such an initiative before it's even started.
There's some amount of government support of medical research:
https://www.nih.gov/grants-funding
It apparently contributes something or other to the system:
http://www.pnas.org/content/early/2018/02/06/1715368115
(annual NIH grants are ~$32 billion, NIH funded research contributed to every drug brought to market in the US between 2010 and 2016...)
https://www.nih.gov/grants-funding
It apparently contributes something or other to the system:
http://www.pnas.org/content/early/2018/02/06/1715368115
(annual NIH grants are ~$32 billion, NIH funded research contributed to every drug brought to market in the US between 2010 and 2016...)
There is a huge gap between "this does X in a petri dish" and a drug that works in the clinic.
I think she means help in the final steps of drug approval and production.
Pair this with Taleb's views on globalism, and how it enables this kind of interconnected environment, that doesn't leave much room for error. And if an error happens, it's going to be systemic.
I like to think of organizations as large living things. Just like people are made out of numerous living cells, organizations are living things made out of numerous living people.
As such, they behave similarly to other living things is some ways. And all living things want to keep living.
As such, they behave similarly to other living things is some ways. And all living things want to keep living.
Really there are quite a lot of things that work like this. It's a tautological truth that things able to influence their own survival will tend toward doing so, since the one's that don't won't survive and are therefore selected against.
Organizations, societies, cultures, nations, and identities, all memetic lifeforms.
Organizations, societies, cultures, nations, and identities, all memetic lifeforms.
e.g. Tax preparation companies lobbying to prevent the tax authorities from supplying candidate computer generated tax returns for tax payers.
Let's just keep in mind that while we observe this in others we probably play the same game ourselves.
Lately I have developed doubts that a system I was part in building up over the last few years and is generally well regarded is still necessary. It's pretty hard to admit this in public if a potential consequence may be to lose your job. I think people would be less prone to preserving problems if they wouldn't potentially lose their job and livelihood.
Lately I have developed doubts that a system I was part in building up over the last few years and is generally well regarded is still necessary. It's pretty hard to admit this in public if a potential consequence may be to lose your job. I think people would be less prone to preserving problems if they wouldn't potentially lose their job and livelihood.
> I think people would be less prone to preserving problems if they wouldn't potentially lose their job and livelihood.
I agree with that. And I think part of the problem is the cultural notion that everything has to be "efficient". But a trade-off must be made between efficiency and robustness.
I think it's a big myth that free market is efficient. No, free market is ridiculously redundant, but what actually makes it working so well is its robustness resulting from the redundancy.
Hierarchical planning, like in big companies, is actually efficient. Global optimization. But being too efficient at the expense of robustness is what leads to unexpected disasters. Being ultra-efficient means no margin for failure. And unfortunately, the executives in big companies are motivated to take risk and overshoot the desired efficiency.
And this idea came to public policy, where we don't want the public money to be "wasted", so we would rather have unemployment than people who are redundant for one reason or another. And that's what the basic income (as other commenter noted) debate is about, essentially.
In the other comment here I also suggest 20% innovation time as a solution. It's actually the same problem - innovation time is seen as wasted, because it doesn't increase efficiency, only robustness.
I agree with that. And I think part of the problem is the cultural notion that everything has to be "efficient". But a trade-off must be made between efficiency and robustness.
I think it's a big myth that free market is efficient. No, free market is ridiculously redundant, but what actually makes it working so well is its robustness resulting from the redundancy.
Hierarchical planning, like in big companies, is actually efficient. Global optimization. But being too efficient at the expense of robustness is what leads to unexpected disasters. Being ultra-efficient means no margin for failure. And unfortunately, the executives in big companies are motivated to take risk and overshoot the desired efficiency.
And this idea came to public policy, where we don't want the public money to be "wasted", so we would rather have unemployment than people who are redundant for one reason or another. And that's what the basic income (as other commenter noted) debate is about, essentially.
In the other comment here I also suggest 20% innovation time as a solution. It's actually the same problem - innovation time is seen as wasted, because it doesn't increase efficiency, only robustness.
Another argument for universal income.
Could this apply to social movements too?
It's basic systems theory. If you want to get really disillusioned, read one of Niklas Luhmann's many books on the matter.
More or less a corollary of Pournelle's Iron Law of Bureaucracy.
H&R Block is the egregious example, with their lobbying against making tax returns simpler for taxpayers, their customers.
Another aspect of this is that an organization will create busy work for itself when none is needed.
E.g. the IT or HR department brain farting new things that solve no problem and nobody asked for just because they can't just sit idle.
E.g. the IT or HR department brain farting new things that solve no problem and nobody asked for just because they can't just sit idle.
Wait, there is a new system? This needs some context.