Big Tech Sees Like a State (2020)(thediff.co)
thediff.co
Big Tech Sees Like a State (2020)
https://www.thediff.co/archive/big-tech-sees-like-a-state/
34 comments
> The more I encounter other powerful government bureaucracies (like building codes, local government, zoning, legislation), the more I suspect they work the same way.
It depends. Knowing some government organizations in Germany this is quite spot on especially when "CEO" level positions are set by political policies based on regular elections and not based on normal carrer-based government servants. However there are exceptions but they tend to be rarer the bigger an organization becomes.
> Somebody at the time it was formalized decided "this is how it should work" on the basis of the best information available at the time, and then once it's formalized, it's virtually impossible to change.
This is indeed also accurate but depends on precedent and the number of relevant dependent entites. If you have precendet of changing a set of rules (somewhat) regularly then dependants can get used to adapting it, e.g., tarifs, tax code and parts of building code.
This however gets a lot harder when more people depend on it because you will then have to make sure (information of/about) your rule changes get propagated appropriately, e.g., updates to driving safety legislation that affect everyone with a car.
It depends. Knowing some government organizations in Germany this is quite spot on especially when "CEO" level positions are set by political policies based on regular elections and not based on normal carrer-based government servants. However there are exceptions but they tend to be rarer the bigger an organization becomes.
> Somebody at the time it was formalized decided "this is how it should work" on the basis of the best information available at the time, and then once it's formalized, it's virtually impossible to change.
This is indeed also accurate but depends on precedent and the number of relevant dependent entites. If you have precendet of changing a set of rules (somewhat) regularly then dependants can get used to adapting it, e.g., tarifs, tax code and parts of building code.
This however gets a lot harder when more people depend on it because you will then have to make sure (information of/about) your rule changes get propagated appropriately, e.g., updates to driving safety legislation that affect everyone with a car.
> Could you use blockchains to essentially create an "artificial executive", a plan of record that everybody gets behind that doesn't fall prey to human fallibilities?
Yes but it will either be a rigid bureaucracy, or subject to manipulation by interested parties, or an inscrutable AI, or some combination of those.
It's not like we have good results when trying to write flawless code, that never needs big fixes, and that is so future proof as to never need updates.
Yes but it will either be a rigid bureaucracy, or subject to manipulation by interested parties, or an inscrutable AI, or some combination of those.
It's not like we have good results when trying to write flawless code, that never needs big fixes, and that is so future proof as to never need updates.
Probably true, but the big question is "where is the efficient frontier, and is it better than what we can achieve now with human institutions?"
I like to think of AI not as "artificial" intelligence but as "aggregate" intelligence. Computers are really good at aggregating the opinions of millions of individual humans. That's what search algorithms like PageRank do, it's what recommendation systems do, it's what voting systems do, it's what markets do, it's what statistical quant trading systems that try to predict the movements of markets do. For these purposes, a computer seems to be reliably better than humans. So why not use computers for what they're good at, aggregating the behavior of millions of individual humans, while still letting humans give their own input.
I like to think of AI not as "artificial" intelligence but as "aggregate" intelligence. Computers are really good at aggregating the opinions of millions of individual humans. That's what search algorithms like PageRank do, it's what recommendation systems do, it's what voting systems do, it's what markets do, it's what statistical quant trading systems that try to predict the movements of markets do. For these purposes, a computer seems to be reliably better than humans. So why not use computers for what they're good at, aggregating the behavior of millions of individual humans, while still letting humans give their own input.
While computers are indeed very efficient at this: to the extent that the aggregation is useful, it gets gamed.
PageRank was great, until SEO was invented by someone who realised there was an opportunity if they gamed it for free advertising, then it became a Red Queen race between SEO teams and Google.
Markets are mostly pretty good, but the USA has a special law about onion futures because someone realised they could game the free market: https://en.wikipedia.org/wiki/Onion_Futures_Act
Etc.
PageRank was great, until SEO was invented by someone who realised there was an opportunity if they gamed it for free advertising, then it became a Red Queen race between SEO teams and Google.
Markets are mostly pretty good, but the USA has a special law about onion futures because someone realised they could game the free market: https://en.wikipedia.org/wiki/Onion_Futures_Act
Etc.
But powerful humans are also gamed, and usually more easily than computers. Haven't you ever told an exec what they wanted to hear, or refrained from telling a manager something that might make you or your project look bad? The whole lobbying apparatus is about altering the perceptions of lawmakers so that you get more favorable laws. People usually operate on fewer pieces of data than computers, and so one falsehood can alter their decision-making more than it would a computers.
It doesn't have to be perfect, it just has to be better than what we have now.
It doesn't have to be perfect, it just has to be better than what we have now.
> But powerful humans are also gamed, and usually more easily than computers.
It's the "more easily" that I'd dispute. A computer can be examined in detail, its weaknesses probed ruthlessly and systematically until a vulnerability is found.
Any one human will have traits in common with others, and differences from those others. For example, your own question:
> Haven't you ever told an exec what they wanted to hear, or refrained from telling a manager something that might make you or your project look bad?
this would be a dangerous similarity, one that can be manipulated — but not universally, not all of us, and not all the time: there's people who react very negatively to such sycophancy or boasting.
> The whole lobbying apparatus is about altering the perceptions of lawmakers so that you get more favorable laws. People usually operate on fewer pieces of data than computers, and so one falsehood can alter their decision-making more than it would a computers.
Yup, absolutely. But if the software developer misses something that an outsider can learn about — this can be something that just didn't even exist at the time the developer made their choices, or just a bug they didn't spot — it can be abused in the same way.
When it comes to accidents, the USA almost started WW3 because someone forgot to account for the fact that the moon rises in the same part of the sky that a Soviet strike group would come from, and also doesn't respond to IFF pings. The Soviet union made their own mistake with sun glare.
When it comes to things outsiders learn and the original developer didn't think of, other than all bugs and exploits, consider also things like this: https://arxiv.org/pdf/1707.07397
It's the "more easily" that I'd dispute. A computer can be examined in detail, its weaknesses probed ruthlessly and systematically until a vulnerability is found.
Any one human will have traits in common with others, and differences from those others. For example, your own question:
> Haven't you ever told an exec what they wanted to hear, or refrained from telling a manager something that might make you or your project look bad?
this would be a dangerous similarity, one that can be manipulated — but not universally, not all of us, and not all the time: there's people who react very negatively to such sycophancy or boasting.
> The whole lobbying apparatus is about altering the perceptions of lawmakers so that you get more favorable laws. People usually operate on fewer pieces of data than computers, and so one falsehood can alter their decision-making more than it would a computers.
Yup, absolutely. But if the software developer misses something that an outsider can learn about — this can be something that just didn't even exist at the time the developer made their choices, or just a bug they didn't spot — it can be abused in the same way.
When it comes to accidents, the USA almost started WW3 because someone forgot to account for the fact that the moon rises in the same part of the sky that a Soviet strike group would come from, and also doesn't respond to IFF pings. The Soviet union made their own mistake with sun glare.
When it comes to things outsiders learn and the original developer didn't think of, other than all bugs and exploits, consider also things like this: https://arxiv.org/pdf/1707.07397
Some DAO’s (decentralized autonomous organizations) function well in their small local area of application, like the rest of crypto there’s usually a centralised organisation behind it, and some scams and failed applications obviously
Yes and no. The point of a government bureaucracy is to eliminate arbitrary decision making from the people in the field to the extent possible.
Taken to the extreme, you have the social security administration, which has a manual which essentially prescribes a procedure for any of thousands of scenarios.
Zoning or code compliance requires some level of professional judgement, but the standards and procedures are driven by legislation or policy.
Pushing that stuff to the end user just pushes the complexity of the business rules to the end user.
Because there’s usually multiple political parties or interests and a level of public accountability, governments are good & efficient at delivering what they are mandated to deliver. (What they are asked to deliver is usually the “problem”.) When was the last time that benefits checks weren’t issued?
Commercial entities with similar qualities are also very bureaucratic (ie banks, airlines). Tech companies tend to be pretty bad at internal controls, and engage in a lot of hold my beer.
Taken to the extreme, you have the social security administration, which has a manual which essentially prescribes a procedure for any of thousands of scenarios.
Zoning or code compliance requires some level of professional judgement, but the standards and procedures are driven by legislation or policy.
Pushing that stuff to the end user just pushes the complexity of the business rules to the end user.
Because there’s usually multiple political parties or interests and a level of public accountability, governments are good & efficient at delivering what they are mandated to deliver. (What they are asked to deliver is usually the “problem”.) When was the last time that benefits checks weren’t issued?
Commercial entities with similar qualities are also very bureaucratic (ie banks, airlines). Tech companies tend to be pretty bad at internal controls, and engage in a lot of hold my beer.
Why would the Shareholders want that?
It would be a new set of shareholders, and their motivation is the same as always: "Other people are rich and I'm not."
Install a good ad/tracker blocker like UBlock Origin on your computers and install a DNS content blocker on your phone like DNS66/Rethink DNS (get them from F-Droid if they've been removed from the Play Store).
I have no regrets on robbing the corporate media and user hostile advertising companies of their revenues, while keeping my browsing experience clean and thwarting most trackers.
I go a step further and run GrapheneOS instead of normal Android, so not even Google themselves can track me. Google Play Services is running in a sandbox that makes it behave.
What were dealing with today is much worse than the old banner ads, these things track your every move online and even track the GPS on your phone.
I have no regrets on robbing the corporate media and user hostile advertising companies of their revenues, while keeping my browsing experience clean and thwarting most trackers.
I go a step further and run GrapheneOS instead of normal Android, so not even Google themselves can track me. Google Play Services is running in a sandbox that makes it behave.
What were dealing with today is much worse than the old banner ads, these things track your every move online and even track the GPS on your phone.
I take issue with your characterization as "robbing". The consumer surveillance industry exploits security vulnerabilities in browsers and OSes (including ones they themselves put there), and then leverages the proceeds into minor attacks on users that net a few pennies at a time. Fixing the security vulnerabilities isn't "robbing" these companies, rather it's merely putting things right and preventing them from being able to take what they have been taking. Furthermore, that the surveillance industry has been normalized and even held up as an exemplar of innovation just illustrates the extent of the rot.
I haven't read Seeing Like a State, but I find the list of legibility criteria to be very interesting, purely because of how many don't apply here in the United Kingdom.
> Lives at a particular location, and has an exact address.
A person in the UK may live at any natural number of places, including zero. The NHS has specific provisions for providing healthcare to the homeless. At the other end of the spectrum, landowners with property in different parliamentary constituencies were entitled to vote in all of those constituencies until the passage of the Representation of the People Act 1948.
> Has a specific name.
In the UK, a person has no specific legal name. A person may use any name they wish, including multiple names, without the need for a deed poll or any other legal instrument.
> Earns money in a currency which the government understands, and pays taxes in it.
The UK has a thriving (although regulated) cryptocurrency sector. Tax policy is a little lax, and it's very easy for someone to live in the UK unbanked and untaxed, which is probably a contributing factor to so many irregular immigrants moving specifically to the UK, rather than remaining in the first 'safe country'.
> Was born on a particular date, and can thus be called up for jury duty or conscription.
The UK is good at tracking when and where people were born, and to whom, but someone can slip through the cracks of the system if they put a little effort in to avoid service to the state.
> Doesn’t steal other people’s goods, trespass on their land, injure them, or kill them. (Except as a result of the aforementioned conscription situation.)
Due to reductions in policing budgets, shoplifting is currently rife and, when the value is low, not addressed, but this is a temporary state of affairs. Trespass is generally a civil matter, and is only criminal in specific situations, such as when the trespass is into a home.
> Speaks a language intelligible to the government employees who are responsible for checking all of the above.
The UK's only official language is Welsh, and Welsh only has that status in Wales. Some legislation is written in Norman French.
On the other hand, I grew up in Australia, where that list very much applies, because Australia is a nation that learnt civilisation from prison guards. It's not a bad list - the UK is just a very peculiar place.
> Lives at a particular location, and has an exact address.
A person in the UK may live at any natural number of places, including zero. The NHS has specific provisions for providing healthcare to the homeless. At the other end of the spectrum, landowners with property in different parliamentary constituencies were entitled to vote in all of those constituencies until the passage of the Representation of the People Act 1948.
> Has a specific name.
In the UK, a person has no specific legal name. A person may use any name they wish, including multiple names, without the need for a deed poll or any other legal instrument.
> Earns money in a currency which the government understands, and pays taxes in it.
The UK has a thriving (although regulated) cryptocurrency sector. Tax policy is a little lax, and it's very easy for someone to live in the UK unbanked and untaxed, which is probably a contributing factor to so many irregular immigrants moving specifically to the UK, rather than remaining in the first 'safe country'.
> Was born on a particular date, and can thus be called up for jury duty or conscription.
The UK is good at tracking when and where people were born, and to whom, but someone can slip through the cracks of the system if they put a little effort in to avoid service to the state.
> Doesn’t steal other people’s goods, trespass on their land, injure them, or kill them. (Except as a result of the aforementioned conscription situation.)
Due to reductions in policing budgets, shoplifting is currently rife and, when the value is low, not addressed, but this is a temporary state of affairs. Trespass is generally a civil matter, and is only criminal in specific situations, such as when the trespass is into a home.
> Speaks a language intelligible to the government employees who are responsible for checking all of the above.
The UK's only official language is Welsh, and Welsh only has that status in Wales. Some legislation is written in Norman French.
On the other hand, I grew up in Australia, where that list very much applies, because Australia is a nation that learnt civilisation from prison guards. It's not a bad list - the UK is just a very peculiar place.
>shoplifting is currently rife and, when the value is low, not addressed, but this is a temporary state of affairs.
Remember though, how often in the history of states, the temporary cannily usurps permanence. It's hard to tell what's a brief deviation from the mean, rather than an early glimpse into the new normal.
I have no idea what I'm talking about, either in terms of understanding society nor the UK specifically, but your phrasing tickled my paranoia about that phrase a sort of famous last words for civilizations, lol.
Remember though, how often in the history of states, the temporary cannily usurps permanence. It's hard to tell what's a brief deviation from the mean, rather than an early glimpse into the new normal.
I have no idea what I'm talking about, either in terms of understanding society nor the UK specifically, but your phrasing tickled my paranoia about that phrase a sort of famous last words for civilizations, lol.
Ah interesting. I'm now colliding this thought with how unnerved I was by the tacit acceptance of very visible elements of the surveillance state in UK. I wonder if the state needed different tactics to make the population legible, and so (without express intention) needed to recruit citizens as more willing participants to reach the functional level of legibility...
It's well known that about a third of all traffic speed cameras in the UK are decoys; I assume that something similar goes on with CCTV cameras.
The British state is administered on the principle of maximum effect for least effort and least cost. The Empire was administered in the same way, although, as Lord North discovered, that principle has limits. The surveillance state is very visible because its primary purpose is deterrence, not solving crime at scale. The population isn't made legible; it's made to think that it's legible.
A few years ago, there was a lot of media coverage about 'super recognisers'[0]. This is because CCTV footage has to be reviewed by humans. There's a reality TV series that shows how much human work has to go into turning the information from the surveillance state into actual outcomes on the ground.[1] It's not an everyday thing.
[0] https://en.wikipedia.org/wiki/Super_recogniser
[1] https://en.wikipedia.org/wiki/Hunted_(2015_TV_series)
The British state is administered on the principle of maximum effect for least effort and least cost. The Empire was administered in the same way, although, as Lord North discovered, that principle has limits. The surveillance state is very visible because its primary purpose is deterrence, not solving crime at scale. The population isn't made legible; it's made to think that it's legible.
A few years ago, there was a lot of media coverage about 'super recognisers'[0]. This is because CCTV footage has to be reviewed by humans. There's a reality TV series that shows how much human work has to go into turning the information from the surveillance state into actual outcomes on the ground.[1] It's not an everyday thing.
[0] https://en.wikipedia.org/wiki/Super_recogniser
[1] https://en.wikipedia.org/wiki/Hunted_(2015_TV_series)
The book does talk about various exceptions, and spends a lot of time talking about how exceptions in some way represent alternative power structures. Having said that, I think much of your list while true would be relatively uncommon (for location, name, currency, birth date and crime) though I can't speak to your last point which is surprising to me!
Then why is the UK still so authoritarian about free speech and insisting on policing online activity?
Because the UK is very peculiar?
Indeed, the perennial excuse of every authoritarian; they are "#Not like other countries".
[deleted]
This feels like its framed in the wrong way.
The much simpler explanation is that both governments and big companies are large groups of people. You need to make simplifying assumptions to make large groups of people work together at scale.
That's true of any system at scale. Just consider how people give their servers unique names at small scale and numbers at big scale. Its just how scaling works.
Whether that is a good or bad thing is an entirely different question.
The much simpler explanation is that both governments and big companies are large groups of people. You need to make simplifying assumptions to make large groups of people work together at scale.
That's true of any system at scale. Just consider how people give their servers unique names at small scale and numbers at big scale. Its just how scaling works.
Whether that is a good or bad thing is an entirely different question.
For some reason we think of corporations and governments as different things. but an incorporation is a government, and a government is an incorporation. that is the whole point, that is why a cooperation is formed.
A group of people want to run their enterprise under rule of law and this requires a government. this legal entity we call a corporation. it may be a for profit endeavor. it might be a public good physical area monopoly(a town, county, state, country). It might be a for profit physical area monopoly(these were some of the first licensed corporations, but you don't see it much now days). anyway there are many types of corporations.
Usually to achieve the desired legal protection a license is acquired from a parent corporation that lets you form the government needed to run the corporation. it's corporations all the way down. At the top level they are peers(nominally) and must guarantee their protection the hard way(armed force). Organized crime is an example of an unlicensed cooperation, there is still a government, but the endeavor is run unlicensed, in rebellion to whatever larger entity claims jurisdiction over the area.
A group of people want to run their enterprise under rule of law and this requires a government. this legal entity we call a corporation. it may be a for profit endeavor. it might be a public good physical area monopoly(a town, county, state, country). It might be a for profit physical area monopoly(these were some of the first licensed corporations, but you don't see it much now days). anyway there are many types of corporations.
Usually to achieve the desired legal protection a license is acquired from a parent corporation that lets you form the government needed to run the corporation. it's corporations all the way down. At the top level they are peers(nominally) and must guarantee their protection the hard way(armed force). Organized crime is an example of an unlicensed cooperation, there is still a government, but the endeavor is run unlicensed, in rebellion to whatever larger entity claims jurisdiction over the area.
Hence my view that even private enterprise is inseperable from it's host government. It is my belief any prohibitions on activity that apply to the government should trickle down to corporations it instantiated. Not privileges, but definitely restrictions.
> For some reason we think of corporations and governments as different things. but an incorporation is a government, and a government is an incorporation.
Most corporations have owners, most governments don’t.
Most corporations have owners, most governments don’t.
It depends on how the group was formed and what rule of law it operates under on who the owner is. where owner is defined on who gets to pick the leaders.
One person grows their operation organically from nothing, they then are the owner and get to be the king/dictator.
A group pools their resources to finance the organization, as such they are the owners to each a share according to their amount.
A group decides that all members of the group are the owners without regard to how much they financed it. this is a democracy.
The point being every group of people has an owner.
One person grows their operation organically from nothing, they then are the owner and get to be the king/dictator.
A group pools their resources to finance the organization, as such they are the owners to each a share according to their amount.
A group decides that all members of the group are the owners without regard to how much they financed it. this is a democracy.
The point being every group of people has an owner.
> For some reason we think of corporations and governments as different things. but an incorporation is a government, and a government is an incorporation. that is the whole point, that is why a cooperation is formed
This lens is increasingly the norm in the PoliSci and Management spaces, with Institutionalism and research into administrative capacity being extremely vogue rn.
Economists like Daron Acemoglu, Yasheng Huang, Rakesh Khurana, etc have dug into this.
This lens is increasingly the norm in the PoliSci and Management spaces, with Institutionalism and research into administrative capacity being extremely vogue rn.
Economists like Daron Acemoglu, Yasheng Huang, Rakesh Khurana, etc have dug into this.
Isn't that basically what the article is saying?
The article is using the vocabulary of sociology, where "legibility" in reference to the state is a well-known concept. The article specifically references Seeing like a state and James Scott's other writings, but he's not the only one to think in terms of such concepts. Foucault's idea of the Panopticon is very similar - it's about the state maintaining control through the citizenry always knowing that they're watched - while Habermas's idea of a "lifeworld" ("metis" in this article) which is "colonized by expert systems" is the same phenomena.
The article is using the vocabulary of sociology, where "legibility" in reference to the state is a well-known concept. The article specifically references Seeing like a state and James Scott's other writings, but he's not the only one to think in terms of such concepts. Foucault's idea of the Panopticon is very similar - it's about the state maintaining control through the citizenry always knowing that they're watched - while Habermas's idea of a "lifeworld" ("metis" in this article) which is "colonized by expert systems" is the same phenomena.
That’s the whole point, though. The need to make things interpretable from the center is what it means to “see like a state.” A society doesn’t need to be interpretable to exist. Large groups of people can relate to one another peerwise.
I think https://doi.org/10.1017/9781009159418 has a different framing of the issue. There's a fundamental conflicts between policy implementation and policy effectiveness.
If policy is designed locally or at least allowed to be adapted to the local context, it could be more effective but with the possibility of inconsistency; Otherwise the policy is implemented flawlessly but with the cost of effectiveness because of lacking local context.
Any attempt of increasing legibility means a policy being implemented consistently across different contexts so everyone can assume the same thing regardless of the context, and, as expected, not able to incorporate the complexity of local social orders.
If policy is designed locally or at least allowed to be adapted to the local context, it could be more effective but with the possibility of inconsistency; Otherwise the policy is implemented flawlessly but with the cost of effectiveness because of lacking local context.
Any attempt of increasing legibility means a policy being implemented consistently across different contexts so everyone can assume the same thing regardless of the context, and, as expected, not able to incorporate the complexity of local social orders.
Even the most dysfunctional and abusive states can generally manage a deeper vision for the future than the next quarter's management bonuses.
Funny the concept of “legibility” is on my mind because I’ve been working on content understanding systems and I quickly came to the conclusion that some content was more legible than other content to the system as opposed to better or worse. (E.g. I hate political image memes because my filters can’t look inside the image easily)
Previous discussion:
Big Tech Sees Like a State - https://news.ycombinator.com/item?id=25009130 - Nov 2020 (14 comments)
Big Tech Sees Like a State - https://news.ycombinator.com/item?id=25009130 - Nov 2020 (14 comments)
I work for one of those Big Tech companies, and internally, it is anything but legible. Very consequential decisions are made by a small team that happens to have responsibility for the codebase in question, oftentimes very far down in the org. Culturally at least there is a respect for data and a sincere effort to do the right thing, but when your decisions impact 3 billion people there is no way a human can fully evaluate them. And ironically, a recurring problem for the last 20 years is that the CEO has no idea what's going on within the company. Most records about what's being worked on, why, how far along they are, who's involved, etc. are firmly in the realm of metis, tribal knowledge that's passed down in 1:1s and watercooler conversations.
The more I encounter other powerful government bureaucracies (like building codes, local government, zoning, legislation), the more I suspect they work the same way. Somebody at the time it was formalized decided "this is how it should work" on the basis of the best information available at the time, and then once it's formalized, it's virtually impossible to change.
I'm interested in crypto not for the "get rich quick" or even the "currency" aspect, but for the prospect of replacing corporations and executives. Crypto's big innovation is in creating a shared reality everybody accepts without a centralized authority to enforce that reality. This is exactly the same efficiency advantage that having a single executive gives. Could you use blockchains to essentially create an "artificial executive", a plan of record that everybody gets behind that doesn't fall prey to human fallibilities?