Education research is hard hard hard. Getting clean studies, especially at large sample sizes, is extremely difficult, leading to a lot of ambiguity in results.
That sort of abstraction has always existed, it's just been a matter of hiring experts or labor from other humans. Reality still has a surprising amount of detail. You deal with it by engaging directly, delegating to someone else to engage with, or just brute force your way to a crooked staircase.
When you hire someone to work on the stairs for you, you /hope/ they know what they're doing, especially if you don't have the skills yourself to judge their work. Same for an agent.
Well, you said people can't use things they don't understand.
But I'll take your expanded statement, to include riding a horse, something even older than the engine. We don't understand fully how a horse works -- biology is still a matter of seeing fragments of the whole -- but people had no problem riding and breeding them before the invention of the car, and before the discovery of genetics.
Meanwhile, understanding the math of a thing -- like stock markets, or nuclear bombs -- does not prevent its use from going badly.
Math is useful and beautiful, and a helpful tool for expanding our understanding of the world, but it is not the whole of understanding, or the sole factor in successful application of science to the world .
How many people drive cars without knowing how an engine works? Or make a phone call without knowing how voice compression for a cellular network does it's thing? Or eats food without knowing how it came together from the supply chain?
I got a very different message from this, actually much closer to the problem of incumbent advantage.
The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem.
The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
It's the standard problem of devolving responsibility to individual action: the effect is so diffuse as to be negligible in all but the most extreme cases. In most cases, one wouldn't even be able to correlate change in sales with the boycott action.
On the other hand, actions like picket lines produce a clear relationship between the issue (look at the sign) and the action. Contacting advertisers and shareholders to get them to pull out their funding also has a direct effect, and connects more cleanly the effect to the cause.
And yet, there was no html5 newgrounds. The magic of flash was that it gave a space where a music person, an art person, and a programmer could bang something out. The barrier to entry was comically low, which allowed an absolute explosion of content.
Sometimes good products happen despite bad technical foundations.
"Stakeholder" literally means someone with a stake in the outcome, which is to say, those who will be affected by the decision. That can include a whole range of people+entities, including citizens (as a group) and the companies to be regulated.
Yeah, it's nice to overlay this with Graeber's Debt - there's cyclic chunks of history where no one has little bags of coins to buy the 10ft poles needed to explore the dungeon. (par example.) Sometimes because the monetary system collapses entirely, sometimes because the nobles have hoovered up all the available silver, and don't have the means to make more.
In those periods, people work more on credits and debts, which shade directly into systems of social obligation and caste when extended over time.
(As you note) It's also very historically recent that 'making money' was seen as any kind of reasonable choice for someone with power. The political and merchant classes are typically quite separate (the exceptions prove the rule); the merchants are picking up an under-explored source of power that is mostly uninteresting to the the ruling class.
The thing is... Interactive updates happen, just in a different way than it does for animal brains. The system is updated with new training data more-or-less constantly. Suppose OpenAI (or whoever) collects a week's worth of conversations with up-thumbs and down-thumbs, or rewritten continuations from human operators, then fine-tunes the current version of ChatGPT with that data. That's an interactive update, and learning from experience. It looks mostly nothing like what we humans do... But it does rhyme a little bit!
We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.
Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.
Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?
Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?
I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.