The problem with many of these kinds of works is that they are a product of a provincial SV mindset that is largely unaware of the broader world or anything at all outside of its cultural and economic ecosystem, so it winds up being a super narrow slice of conventional, mostly boring topics that everyone in SV rehashes over and over.
Wouldn't it be interesting to learn an influential work that changed how health care professionals run hospitals? Or a document that changed how mining works? A paper on Wright's theory of manufacturing scaling that explains the solar revolution? A thesis on how the world's factories moved from pneumatics to servo systems and why? How a policy thesis changed federal regulators' approach to approving rare disease drugs? Maybe Hayek's views on socialism and information theory, or perhaps an influential thesis on how antitrust monopoly regulation should work?
But instead it's a bunch of crypto and AI stuff that everyone in tech already knows about, rewarmed again for its hundredth serving.
That's annoying, but it's not why scaled manufacturing is lowering unit costs of panel production. Look at bare panel prices, they've followed the same cost curve down.
The same problem exists in the airline market. Airline ticket prices are historically very low, but people complain about seats, fees, and so on. But then they keep buying the absolute cheapest ticket.
What consumers say they care about, and what they actually care about, are not the same. Otherwise they'd pay more for the less irritating product.
I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI.
With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.
With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.
In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.
That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.
Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.
Bender's paper had this to say about stochastic parrots:
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
Regulatory capture doesn't necessarily mean the regulated get to decide what the regulators do in precise steps. It can simply mean they support and exist within a regulatory regime that greatly benefits the regulated.
In fact, you generally don't want them directly telling the regulators what to do. Instead, the regulators make complex, costly rules that only large establishment players can follow. The regulators look like they're doing their job; the regulated enjoy higher margins and protection from disruption.
He has this wealth as a function of having sold the equity in the company he co-founded, so in fact virtually all his wealth has already gone through a tax event at the federal and state levels.
On-prem versus cloud inference doesn't matter for concentration of power.
Concentration of power exists when the model makers are the same as (or control) the inference providers. Making a model is capital intensive, so there aren't many of them. Providing inference is not: I don't even need to own GPUs; I can rent them from those who do and then sell by the token. B300s cost less than $4 an hour currently.
Cloud can even be more effective at lowering concentration of power than on premise. Asking people to individually buy $20,000 of compute equipment plus power and cooling equipment to run a frontier model is not something they're going to do if they can just pay four-tenths of a cent per output token. If the only cloud inference providers are the big proprietary US titans, that means you're going to get far more power concentration than if open source inference providers are an alternative, because then I can just switch my API endpoint.
Yes, people conflated progressive disclosure as a method with skills as a particular implementation because skills became the first widely adopted use of progressive disclosure.
But progressive disclosure is just a method that you can apply to lots of things to reduce context bloat. Any time you provide some kind of limited index or search to an AI and then let it expand that based on the circumstances of the request, it's progressive disclosure.
And one of the things you can apply it to is MCPs.
The government would be far better off figuring out how to take commodity models and applying them to government functions where they can, with deterministic scaffolding and guardrails, to make government more efficient, optionally using RL on traces from their use to improve their performance.
Imagine taking models and fine-tuning them / doing RL rollouts to help automate permit application approvals, as applied specifically to Dutch permit processes. That would be a real help to Dutch businesses!
That type of applied AI is more interesting and effective now than just trying to make another foundational model that isn't going to work well or do anything of economic value.
It's right that they steer toward the center of their distribution. But I would offer a different view on whether that's a step up for half the population.
Writing isn't a distribution on a single dimension that goes from "bad" to "good". It's a lot of dimensions that encompass everything from "funny" to "formal", "precise" or "hysterical". They may be filled with metaphors, or use allegory; they may use math or logic to explain. The allegories could be from science fiction or they could be Biblical or 19th century Victorian novels. None of these are right or wrong, but they are opinionated ways to express an idea.
Writing feels better when it has real texture and character to it. That character is not the monodistribution of "bad" to "good". It's whether it inhabits pockets of out-of-distribution thought in the thousands of dimensions of "thought-space."
An LLM pushing to the center of distribution means it pushes the writing out of inhabiting any of the interesting pockets that create the feeling of texture. The middle of the distribution does not mean it is average quality: it means it's not good at all. The median of the distribution can be far worse than the median writer if you accept that the median writer has out-of-distribution thoughts on at least something, and that it is this which makes their thoughts interesting.
That's why a rough, perhaps not-totally-grammatical article written by someone with interesting thoughts is vastly better than a "correct" LLM revision, even if the human writer isn't a 'good' writer. Their article occupies an opinionated stance on some dimension that matters; it sits in a pocket of interestingness that LLMs seem almost totally unable to inhabit.
The exact middle of the distribution across thousands of dimensions may actually be one of the very worst places of them all.
I can't stand it when LLMs tell you to tone it down. Their writing advice is almost universally awful. They want you to write the most cliched bland content possible.
Sometimes I see technical people who feel they aren't good writers, but who have good ideas. They then turn to LLMs, believing that the LLM will help them express their good ideas. They're often right that they have good ideas. But the LLM just turns them to sawdust.
Kudos to spurning the mediocrity conversion machine and hitting publish.
Anthropic is, I believe, fully pursuing the idea that you shouldn't use their model with anything but their own products. They don't care whether it generalizes.
I agree it's very frustrating to use with custom tools/harnesses that can speed up the process for domain specific purposes.
That is not what their policy states. It specifically says they will sabotage even non-distillation attempts, such as distributed training pipeline design. And given that they are so far very nonperformant in classification accuracy, expect it to randomly include far more topics wide of the mark.
The fun part is that you will never know if your neural net classification project is getting silently sabotaged because their classifier doesn't work!
Wouldn't it be interesting to learn an influential work that changed how health care professionals run hospitals? Or a document that changed how mining works? A paper on Wright's theory of manufacturing scaling that explains the solar revolution? A thesis on how the world's factories moved from pneumatics to servo systems and why? How a policy thesis changed federal regulators' approach to approving rare disease drugs? Maybe Hayek's views on socialism and information theory, or perhaps an influential thesis on how antitrust monopoly regulation should work?
But instead it's a bunch of crypto and AI stuff that everyone in tech already knows about, rewarmed again for its hundredth serving.