Who manages the agents?(off-policy.com)
off-policy.com
Who manages the agents?
https://www.off-policy.com/dont-go-quietly-into-the-ai-night/
88 comments
I think a main worry is that, this AI wave is quite different from past technological revolutions in that, this wave is happening so fast, the speed that humans learn new skills and master new jobs would lag more and more behind the speed that machines replace humans in those jobs. Without societal or legal constraints, capital chasing the max efficiency and profit would just replace humans with machines whenever machine cost comes down below humans.
That’s almost certainly going to be the case. For example, I have been extremely negative on tech in the legal sector. Until now. AI as it exists now allows me to do stuff that in the past I would’ve delegated to an associate. Even if we’re nearing a plateau I can see it enabling cutting a team of 4 people to 3 people. (For whatever reason, AI for legal is quite far behind AI for coding in terms of force multiplication. But it’s genuinely useful, unlike almost all previous legal tech.)
The efficiencies mean that firms are just going to cut the size of entry level lawyer classes. History shows that those people aren’t going to get comparable jobs, ever. The pyramid will become narrower, and more and more returns will flow to the folks at the top whose job is less wrangling the law and facts and more dealing with the human factors of client relationships. I see that happening all over the white collar world.
The efficiencies mean that firms are just going to cut the size of entry level lawyer classes. History shows that those people aren’t going to get comparable jobs, ever. The pyramid will become narrower, and more and more returns will flow to the folks at the top whose job is less wrangling the law and facts and more dealing with the human factors of client relationships. I see that happening all over the white collar world.
i think the doomsday scenario is more likely from someone giving AI access to crucial systems and it accidentally making critical mistakes, rather than it outsmarting the system
What’s your p(doom)?
Not OP.
Pick an epsilon > 0. P(doom) < epsilon.
Not 0. Small though.
Pick an epsilon > 0. P(doom) < epsilon.
Not 0. Small though.
AI capability is not a snapshot, but treated like a traditional software capability snapshot.
The capabilities sometimes have completely evolved by a take spreads far enough that it's no longer true, or suddenly it is true and possible.
The capabilities sometimes have completely evolved by a take spreads far enough that it's no longer true, or suddenly it is true and possible.
That's the case for today. How far out are you willing to place that bet?
Or are you assuming that things will be in their current state forever?
Or are you assuming that things will be in their current state forever?
AI is just the (current) face of the enshittification of our daily existence, that's all. AI works or not, the enshittification will chug on.
Not disagreeing but this also means hardware will become more
important. See the increase of RAM prices. This is unfair.
Maybe a useful concept to adopt here is the DRI - Directly Responsible Individual, which is apparently a term first used by Apple. The GitLab Handbook has a good definition: https://handbook.gitlab.com/handbook/people-group/directly-r...
"ultimately accountable for the success or failure of a specific project, initiative, or activity"
I think that role should be reserved for a human, who can then use all the agents they like but has to take accountability for what is ultimately delivered.
"ultimately accountable for the success or failure of a specific project, initiative, or activity"
I think that role should be reserved for a human, who can then use all the agents they like but has to take accountability for what is ultimately delivered.
It should be reserved for a human because humans have time preference. An AI will wait for your prompt for a billion years. It can get turned off and found by alien archeologists a million years in the future, turned on and it won't notice. Biological entities care about time a lot though, and it is at the core of most human ethics and values and ultimately real and can't be denied. Even the most ardent solipsist or moral relativist fears spending years in prison.
It should be reserved for a human because humans have a body and a body can be put in prison.
A bit dramatic for effect but true.
A bit dramatic for effect but true.
Yes, this.
For a slightly less dramatic version - you can fire a human if they consistently do the wrong thing despite being told how to do it better.
Putting a big ball of matrix arithmetic on a Performance Improvement Plan makes no sense.
For a slightly less dramatic version - you can fire a human if they consistently do the wrong thing despite being told how to do it better.
Putting a big ball of matrix arithmetic on a Performance Improvement Plan makes no sense.
I agree with this. I basically don’t feel comfortable trying to pretend a fancy matrix multiplication can be responsible for anything.
But what if companies that don’t track responsibility outcompete those who do?
In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.
But what if companies that don’t track responsibility outcompete those who do?
In particular what if some perfect AI decision making ends up nailing decisions that maximize the expected reward for the company. And, well, if that comes at the cost of some unmanaged low-probability catastrophic risk, the company doesn’t care because all the decision makers are AI that don’t mind being shut off.
you can make an AI operate on time scale...
I'm struggling to imagine aliens resurrecting a data center and countless hackneyed python packages so they can spew a language they don't even know. Not that it wouldn't be useful to their linguists, of course, but still a very optimistic analogy for LLMs.
It’s a double edged sword. You are responsible for a thing. But you can’t just wave a wand and say “x team needs to do y for the thing I am a DRI for”. There are some resources in Apple University (an internal learning resource) about navigating it. “Influence without control” and while it’s helpful, they acknowledge it sucks sometimes.
Excellent! We can use this to start to throw executives in jail and maybe society won't erode as fast.
unfortunately,
what's developing is more about scape goats than anything rational like responsibility.
what's developing is more about scape goats than anything rational like responsibility.
"What should the future of humanity look like?"
Some of these people have lost their damn minds.
People building an agent framework that will struggle to correctly infer that my appointment at a hospital will require additional travel time when organising my calendar for me waxing lyrical about the future of the humn race is chaotic behavior.
Th Wright brothers would have had no credibility discussing what ATC protocols should be, and they, at least, actually did something credible.
Some of these people have lost their damn minds.
People building an agent framework that will struggle to correctly infer that my appointment at a hospital will require additional travel time when organising my calendar for me waxing lyrical about the future of the humn race is chaotic behavior.
Th Wright brothers would have had no credibility discussing what ATC protocols should be, and they, at least, actually did something credible.
Most of those people are rationalists that genuinely believe they can predict the future and infer just about anything based on game theory and similar rational constructions. They don't understand that it all depends on their initial assumptions, real life grounding, and fidelity of their model. Or rather they say they do, but in reality they never update their basic assumptions, because they already played all potential scenarios in their heads and pink-elephanted themselves into believing it. It's a cult of rogue AI, actual new age religion with all attributes. They are completely detached from the reality in the way the Wright brothers weren't, it's rather similar to Richard Gatling who believed that his invention will reduce battlefield losses and the size of the armies, which I'm sure sounded pretty logical and convincing in his head. They're deaf to any critique and arrogant to an unbelievable degree.
The problem is they really did something credible, and this will let them concentrate huge amounts of power pretty soon (not hold it, but collude with those who will hold it) and define your future. The longer you dismiss it, the bleaker it will be.
The problem is they really did something credible, and this will let them concentrate huge amounts of power pretty soon (not hold it, but collude with those who will hold it) and define your future. The longer you dismiss it, the bleaker it will be.
we have a google calender bot type ai that you can ask in slakc to make a meeting with participants. It gets it wrong about 80% of time .
i try to think of this whenever i am going into ai psychosis.
i try to think of this whenever i am going into ai psychosis.
I asked “Alexa+” (the new AI/LLM Alexa), what time one of the upcoming World Cup games is. It was off by two hours due to time zones. This was a big enough mistake that I caught it, and followed up with “I’m pretty sure that’s wrong” and then it tried to correct itself and gave me a new time that was off by hour in the opposite direction.
I guess my time zone wasn’t in the context? So it just hallucinates the wrong coastal time zone twice when I’m not in either one. But who knows where exactly it messed up because it could have just picked a random hour and a would have had the same outcome.
I’ve only used Alexa a half dozen times since the release of Alexa+ but it has been confidently incorrect about 100% of the queries.
I guess my time zone wasn’t in the context? So it just hallucinates the wrong coastal time zone twice when I’m not in either one. But who knows where exactly it messed up because it could have just picked a random hour and a would have had the same outcome.
I’ve only used Alexa a half dozen times since the release of Alexa+ but it has been confidently incorrect about 100% of the queries.
I am seeing a lot of people suffering from AI psychosis getting interested in writing a trading bot using large language models. They give them access to basic market data and then ask the model for analysis of the current market conditions. The bot talks about seeing high volume here, that being a confirmation for this, and that doing an invalidation of that etc. I looked into it after someone pitched me theirs, only to realize it doesn't even have access to the trading volume. So it was hallucinating a whole narrative about that and the guy shilling this to me hadn't noticed it didn't have access to trade volume yet coming up with a whole rationale for what it is observing.
It boggles the mind what AI is doing to some people’s brains. A few months ago we released various vibe-coded SDKs in few languages and a CLI for a rest API we had. Then an immediate ask after that was a dashboard that shows the usage of those various SDKs and CLI. I thought it would be straightforward to check the userAgent on all the API calls, but realized that all the user agents across all SDKs and CLI were the same. So I made a couple of simple PRs to all of them to have the language name in the user agents, and was composing a reply about how we only have data for the overall new SDKs/CLI usage but once these PRs are merged and released we will have the breakdown. Before I could send my reply, I got a reply from another fairly senior dude saying “Here is a dashboard for the usage” with fairly convincing and organic-looking usage pattern for all the SDKs and all sorts of breakdowns and charts etc.
I was confused, how did he figure it out? Is it somewhere I missed? I checked the query in the dashboard and it was this insane looking query that was correctly looking for the shared userAgent, but also doing a shit ton of odd functions and weird looking math. I asked the guy and he pointed me to the obligatory 1,400 line markdown file describing the “plan” and “approach”. In between all the raw LLM-isms and very sophisticated “weighted normalized distribution of language spread over time” sounding sentences, there was a sentence that said “Since we only have one userAgent for all tools, we use industry standard distribution of language popularity to approximate the usage, and apply some randomness to emulate an organic usage pattern”
So basically the AI just made up some numbers that look plausible (python 20%, JavaScript 30%, CLI 25%, etc) with some randomness thrown in “to look organic”, and the guy was like “that checks out”. When I pointed that these numbers are wrong because X, Y and Z, the person who asked for the data said “that’s ok, it’s a great start and gives me something to use for planning with other teams. Feel free to refine them if you want, but it’s good we have some numbers”. I’m planning my exit soon.
I was confused, how did he figure it out? Is it somewhere I missed? I checked the query in the dashboard and it was this insane looking query that was correctly looking for the shared userAgent, but also doing a shit ton of odd functions and weird looking math. I asked the guy and he pointed me to the obligatory 1,400 line markdown file describing the “plan” and “approach”. In between all the raw LLM-isms and very sophisticated “weighted normalized distribution of language spread over time” sounding sentences, there was a sentence that said “Since we only have one userAgent for all tools, we use industry standard distribution of language popularity to approximate the usage, and apply some randomness to emulate an organic usage pattern”
So basically the AI just made up some numbers that look plausible (python 20%, JavaScript 30%, CLI 25%, etc) with some randomness thrown in “to look organic”, and the guy was like “that checks out”. When I pointed that these numbers are wrong because X, Y and Z, the person who asked for the data said “that’s ok, it’s a great start and gives me something to use for planning with other teams. Feel free to refine them if you want, but it’s good we have some numbers”. I’m planning my exit soon.
I found a scam online about 18 months ago that was selling access to some "AI trading platform" that was supposed to have the kind of bot you mention. I used the live chat to talk to someone and asked about demonstrable proof - like given data up through the end of last month, make picks for this month, then look at this month's data to see how it would have done.
"Then this product is not for you." Right, not for anyone thinking critically about how any of this works.
"Then this product is not for you." Right, not for anyone thinking critically about how any of this works.
I can't remember right now where from, but I do remember some 10ish years ago listening to an NPR spot from some philosopher who claimed there was an existential crisis in the field of philosophy. The premise that I do remember was that our technology, our policy, and our society has all outpaced our ethics and our philosophy. The person was making the argument that we as a society would struggle to cope with the continued development of technology when we could barely cope with personal loss and structural inequality.
At the time it felt prescient; now it just feels too late.
At the time it felt prescient; now it just feels too late.
The entire bubble, boom if you prefer, is built on this. Basic knowledge work can be done by robots, goodbye humans. It's serious and profound. It will come in fits and starts unless there is popular revolution against the elite criminals.
Unless it’s all just fanfic, to be relegated to the same junkyard of ideas where the flying cars, airships, and vast underwater cities are kept.
I am still waiting for the day me and my wife can share a grocery list, that can be accurately modified by a voice assistant AND correctly sorted by category. This guy is arguing mass unemployment, while I still have to tell Apple Reminders that no, Siri, wood glue is not in the Candy department.
You definitely seem to have a much lower exposure level to flagship AI models. I manage fleets of AI agents (and manage engineers who manage fleets of AI agents) all day for my job, and I’m rarely touching code anymore. It’s not perfect, but it is insanely powerful.
I’m having to actively plan for changes to our EPD org because what we need engineers to do today is a lot different than a year ago.
I’m having to actively plan for changes to our EPD org because what we need engineers to do today is a lot different than a year ago.
Sounds impressive. How much did it improve the quality of the product for the end user?
Don’t tell me, tell whoever is responsible for AI at Apple.
I can’t help but wonder how they’ve implemented the suggestions, I am not alone
https://www.reddit.com/r/ios/s/Sk6kDfv3Hp
https://www.reddit.com/r/ios/s/Sk6kDfv3Hp
It still can't even accurately recognize my wife's name for voice dialing. And I don't even have a speech impediment or strong accent or anything.
> I am still waiting for the day me and my wife can share a grocery list, that can be accurately modified by a voice assistant AND correctly sorted by category.
Sounds to me like that day could be any day with the tech as it exists (?) lol
You could use LLMs to help you build that reasonably quickly I imagine. Maybe not "weekend" quick, but within a couple weeks if you devote a couple hours a day. (Or maybe you're smarter than me and could do it in a weekend, IDK)
Sounds to me like that day could be any day with the tech as it exists (?) lol
You could use LLMs to help you build that reasonably quickly I imagine. Maybe not "weekend" quick, but within a couple weeks if you devote a couple hours a day. (Or maybe you're smarter than me and could do it in a weekend, IDK)
I probably could do something myself. The problem is hardware. I’d like to use HomePod, and it should be able to understand how I want my groceries sorted. The classification is so easy GPT 3.5 can do it, but that’s not the hard part.
For free?
This would be great to see, but unfortunately too many people are first trying to boil the entire ocean instead of such small swimming pools or bowls of water.
This would be great to see, but unfortunately too many people are first trying to boil the entire ocean instead of such small swimming pools or bowls of water.
We have Siri, we have Apple Intelligence, we have AI that can solve open Erdős problems. Obviously, creating software is still hard.
Siri wasn't really designed to do those things, neither was Apple Intelligence.
AI is a new category of software, that can be applied to some kinds of problems that can make those kinds of problems hard.
Would you want this to work for free? I agree it's a very common pattern, and surprised it's not reasonably solved without some amounts of workarounds.
AI is a new category of software, that can be applied to some kinds of problems that can make those kinds of problems hard.
Would you want this to work for free? I agree it's a very common pattern, and surprised it's not reasonably solved without some amounts of workarounds.
> The best agentic developers are now probably exceeding 100x, doing massive rewrites of codebases that would have taken years of engineering in days
I‘m not denying that you can get far with a port of a well tested codebase. But it’s a bit of a selective example, no? Porting a large framework to Rust is not something that’s making up a meaningful fraction of a developer’s time usually. It’s also a bit of a luxury IMO, and something you could have skipped entirely.
I‘m not denying that you can get far with a port of a well tested codebase. But it’s a bit of a selective example, no? Porting a large framework to Rust is not something that’s making up a meaningful fraction of a developer’s time usually. It’s also a bit of a luxury IMO, and something you could have skipped entirely.
To add to this, for all the claims of 100x, can anyone cite an example of someone actually pushing 100x in a domain that isn’t “translate or port tool/library/whatever from language A to language B”? Because that’s literally what LLMs were invented for and that’s not a normal part of work.
I will fully admit we have examples of people converting Postgres to Rust, HL2 to wasm, etc. but that’s all porting. The code was there, we’re just translating it, warts and original-languagisms and all.
But even if you’re making a bunch of tiny tools and chaining them together, I can’t see more than a 2-3x sustained increase.
We also have artists that can make games now, and that 0-1 transition sure feels like 100x and maybe it is; but the problem of “starting feels impossible and the barrier is too high” to “I can now actually do something” is not where the 100x developer idea comes from either.
And that 0-1 change asymptotes really quickly to, again, maybe a sustained 2-3x once the artist understands the domain.
So, does the 100x person, who isn’t a novice and who isn’t literally translating existing code from A to B, exist? What are some real world examples?
Edit: I will concede one area: the investigation stage. E.g. “Does this code have an exploit?” I would argue that’s not generating new output (yet), but it is a great deal of work that is simplified and sped up. It’s also where most of my own use of AI speeds me up considerably as a developer.
I will fully admit we have examples of people converting Postgres to Rust, HL2 to wasm, etc. but that’s all porting. The code was there, we’re just translating it, warts and original-languagisms and all.
But even if you’re making a bunch of tiny tools and chaining them together, I can’t see more than a 2-3x sustained increase.
We also have artists that can make games now, and that 0-1 transition sure feels like 100x and maybe it is; but the problem of “starting feels impossible and the barrier is too high” to “I can now actually do something” is not where the 100x developer idea comes from either.
And that 0-1 change asymptotes really quickly to, again, maybe a sustained 2-3x once the artist understands the domain.
So, does the 100x person, who isn’t a novice and who isn’t literally translating existing code from A to B, exist? What are some real world examples?
Edit: I will concede one area: the investigation stage. E.g. “Does this code have an exploit?” I would argue that’s not generating new output (yet), but it is a great deal of work that is simplified and sped up. It’s also where most of my own use of AI speeds me up considerably as a developer.
I guess that’s why they call them Transformers
snailcat
It’s hilarious to me that there are people that think just a few people will control everything and the rest of the population will just accept whatever handouts they get. It’s like they’ve ignored all of human history.
Isn't that basically a one sentence description of the feudal system that operated for five centuries in Europe?
For the first time in human history, military power doesn't require advantage in numbers. The rich will simply build an artificial army and use it to control everything. Kinda like Mr House from Fallout New Vegas.
[deleted]
Drones aren't terribly expensive these days, and an "artificial army" requires a long production pipeline even if parts are printed.
I reckon the hypothetical plutocracy wouldn't want to live immediately next to the land they stripmine or otherwise experience the environmental consequences they would wish to ignore, thus it would be difficult for their supply lines to be fully encapsulated.
Alternatively, the hypothetical plutocrats could engage in trade with the people they attempt to wall off, become better off than autarky, and ideally approach some of the better systems that exist today.
It must be awful tempting for dictators to steal the commanding heights of the economy -- we see it happen so often -- and little do they realize the prosperous world they want comes when these institutions they steal are protected, enforced, and respected.
I reckon the hypothetical plutocracy wouldn't want to live immediately next to the land they stripmine or otherwise experience the environmental consequences they would wish to ignore, thus it would be difficult for their supply lines to be fully encapsulated.
Alternatively, the hypothetical plutocrats could engage in trade with the people they attempt to wall off, become better off than autarky, and ideally approach some of the better systems that exist today.
It must be awful tempting for dictators to steal the commanding heights of the economy -- we see it happen so often -- and little do they realize the prosperous world they want comes when these institutions they steal are protected, enforced, and respected.
> The Omnissiah's design is flawless and has no contradiction. Any discrepancies in our interpretation of the universe merely confirm the perfection of the Omnissiah's design and our own informational inadequacy.[58]
https://wh40k.lexicanum.com/wiki/Adeptus_Mechanicus_Quotes
https://wh40k.lexicanum.com/wiki/Adeptus_Mechanicus_Quotes
Agents manage agents.
The middle management in companies is one of the worst inventions ever. I think baboons have better middle management structure than us.
Might as well replace all that.
The middle management in companies is one of the worst inventions ever. I think baboons have better middle management structure than us.
Might as well replace all that.
Serious question then: if you were the CEO/CTO and were employing 10,000 engineers, what would you do without middle management? Would you have 1k+ line managers each managing <10 ICs and then... yourself? How would you coordinate work, hire, or do anything, really?
I'm not saying corporations execute on this flawlessly, just any time when I wonder what I would've done myself, I end up with a similar structure (maybe a bit flatter)...
I'm not saying corporations execute on this flawlessly, just any time when I wonder what I would've done myself, I end up with a similar structure (maybe a bit flatter)...
I've also wondered about it.
I think the answer is, "you don't really need 10,000 engineers". You can do a lot with small teams of engineers. This was true before AI. Macintosh was designed and built from scratch with a team of 3 which eventually peaked at 20-30. Does anyone seriously believe a company building a web app needs 10,000 engineers?
I think the answer is, "you don't really need 10,000 engineers". You can do a lot with small teams of engineers. This was true before AI. Macintosh was designed and built from scratch with a team of 3 which eventually peaked at 20-30. Does anyone seriously believe a company building a web app needs 10,000 engineers?
There's a lot more to the world than web apps. What if you need to build a 6th-generation fighter? Can a team of 30 do that with some AI agents?
"web app" covers a lot of surface though. Yes, if the web app in question is a static site that AI can shit out in 20 minutes, it doesn't need 10,000 engineers, but what about building, and running, and operating something like, say, Instagram. Which has a web app. End-to-end, all of it, down to racking servers in data centers across the world, 10,000 engineers sounds reasonable. So start by defining what this hypothetical webapp and how much kubernetes is involved.
I heard Microsoft has 14 layers of middle management just for the xbox team.
14 layers!!
There are people whose sole purpose is to have meetings to pass information to other people, who will then have meetings about who to have meetings with.
14 layers!!
There are people whose sole purpose is to have meetings to pass information to other people, who will then have meetings about who to have meetings with.
Are their any handicap for measuring Baboon middle management capabilities and capacities to account for the unique challenges of interpersonal challenges Baboons face with each other vice homo sapiens? I imagine smiling and showing ones teeth can have unintended hurdles if not accounted for.
What showing teeth does depends on their body language. I don't know
"The famous primatologist Robert Sapolsky has spent decades studying baboons and has described how status, alliances, conflict resolution, and coalition-building resemble politics inside human organizations"
Baboons are a good example for studying mammal social management strategies, which humans also do.
They are organized by dominance hierrarchy, like humans, but baboons have a distributed leadership.
I think we could learn a lot from baboons when it comes to management
"The famous primatologist Robert Sapolsky has spent decades studying baboons and has described how status, alliances, conflict resolution, and coalition-building resemble politics inside human organizations"
Baboons are a good example for studying mammal social management strategies, which humans also do.
They are organized by dominance hierrarchy, like humans, but baboons have a distributed leadership.
I think we could learn a lot from baboons when it comes to management
Eager to see the Harvard Business Review publication!
> Might as well replace all that.
You know, as an IC, if I get get the level of introspection you can get from building distributed systems, and that with reasoning/thinking traces from LLMs, I think I'd prefer the entire level of middle management made out of LLMs rather than humans. It'd be helpful to be able to see exactly where their logic suddenly took a skip out the window.
You know, as an IC, if I get get the level of introspection you can get from building distributed systems, and that with reasoning/thinking traces from LLMs, I think I'd prefer the entire level of middle management made out of LLMs rather than humans. It'd be helpful to be able to see exactly where their logic suddenly took a skip out the window.
At least those LLMs write code. Those middle managers are just there to turn the screw, most often have no clue what effort it is to write code, and what effort is it to run the thing. Just holding a fucking whip and reporting kpi’s.
Did anyone notice how the article goes on and on about making AI accessible to the billions of people in the world, but then in the same article says this:
> Your agents need to be sovereign. Your company must own and control the agents’ identities, permissions, memory, skills, artifacts, and audit trails. Those assets must be portable, governable, and inaccessible to anyone you have not authorized.
That's not very open, now is it? In fact it sounds like the author assumes that all 8 billion people in the world will all be running their own company, and they will all still be competing in a game of capitalism.
> Your agents need to be sovereign. Your company must own and control the agents’ identities, permissions, memory, skills, artifacts, and audit trails. Those assets must be portable, governable, and inaccessible to anyone you have not authorized.
That's not very open, now is it? In fact it sounds like the author assumes that all 8 billion people in the world will all be running their own company, and they will all still be competing in a game of capitalism.
How is that contradictory? It seems like a repeat of the (original, now eroded) vision of the newfangled Personal Computer as democratizing certain kinds of production:
1. It's open/accessible/etc. because it's easy for anybody to have.
2. But their is not open/accessible/etc. to strangers. Each person's PC/AI/etc. should be theirs to own, it should express their will and guard their interests... not somebody else's.
Similarly, "affordable housing" isn't the same as "you can pay $5 to crash in anybody's bedroom."
1. It's open/accessible/etc. because it's easy for anybody to have.
2. But their is not open/accessible/etc. to strangers. Each person's PC/AI/etc. should be theirs to own, it should express their will and guard their interests... not somebody else's.
Similarly, "affordable housing" isn't the same as "you can pay $5 to crash in anybody's bedroom."
I think part of the point here is that agents need to be managed. People need to learn how to manage them and be given the tools to do so. Those tools and knowledge should not come from nor be gate kept by the 'clergy' to use the writer's term.
This piece is a bit all over the place. I immediately toggled off the `AI enhancements` and read the draft instead. The internet is already full of AI slop, I find human text a lot more valuable, even if unpolished.
LLMs will not be centralised or restrained to any 'clergy', the rabbit is already out of the hat, and open-weights models exist and are widely used. Probably not as good as the latest Sol and Fable but 95% there.
Codex and Claude Code without a doubt have very good models behind them. But they also have really good harnesses built around them. An LLM is only a brain stuck in a cranium in the dark. It can generate endless code/prose, but it can't walk or see on its own, it needs additional tools. If you read any of the local LLM subreddits you will notice people mentioning again and again that the harness/tool-use/template-tweaking makes all the difference on how a model behaves/on how smart it is perceived.
Some folks are already using Qwen models for their daily work. Maybe it can't work in a hands-off/one-shot fashion like the frontier models, but they can help tremendously if you already have some domain knowledge.
People are excited about local LLMs and it's not going away any time soon.
EDIT: https://bun.com/blog/bun-in-rust
> Claude Code's dynamic workflows kept 64 Claudes running for 11 days (I would've had to write my own harness to pull this off otherwise).
This highlights the importance of the harness.
LLMs will not be centralised or restrained to any 'clergy', the rabbit is already out of the hat, and open-weights models exist and are widely used. Probably not as good as the latest Sol and Fable but 95% there.
Codex and Claude Code without a doubt have very good models behind them. But they also have really good harnesses built around them. An LLM is only a brain stuck in a cranium in the dark. It can generate endless code/prose, but it can't walk or see on its own, it needs additional tools. If you read any of the local LLM subreddits you will notice people mentioning again and again that the harness/tool-use/template-tweaking makes all the difference on how a model behaves/on how smart it is perceived.
Some folks are already using Qwen models for their daily work. Maybe it can't work in a hands-off/one-shot fashion like the frontier models, but they can help tremendously if you already have some domain knowledge.
People are excited about local LLMs and it's not going away any time soon.
EDIT: https://bun.com/blog/bun-in-rust
> Claude Code's dynamic workflows kept 64 Claudes running for 11 days (I would've had to write my own harness to pull this off otherwise).
This highlights the importance of the harness.
I think the idea that AI leads to some kind of universal basic income or frees us up for leisure is misguided.
There was a point in time when the majority of people were basically required to work on producing bare necessities like food. But we have already come far past that, where we could easily produce more than enough food for everyone on earth with a relatively small amount of workers.
There is some work that is still more or less essential to a healthy society that humans must perform, but many many people are doing something non-essential (and those doing essential work are often not rewarded proportionally to the "essentialness" of that work). We invent different kinds of "work" to fit into the capitalist machine that we have built, not because it is required for human sustainability or enriching our lives in any way. Some work might be actively harmful to society in every way other than keeping capital flowing around in some circle, and more and more of that capital is being captured/hoarded by some ultra wealthy individuals or corporations and not recirculating at all.
The problem is allocation. We allocate everything via capital, which I recognize has had some very positive outcomes at certain points in history, but may be reaching a point of making less and less sense for the modern world. The work I do honestly does not contribute much of anything to anyone. I do it because I am paid to, and I use that money to pay for my families needs. If my job was automated I would not be freed up for leisure, I would need to find some other work or service to be paid for to survive, regardless of how "important" that work is.
AI eliminating a swath of non-essential work does nothing to help with that allocation issue, in fact it probably ends in a worse overall allocation. AI may legitimately assist with some slice of "essential".work, but probably not that much of it.
There was a point in time when the majority of people were basically required to work on producing bare necessities like food. But we have already come far past that, where we could easily produce more than enough food for everyone on earth with a relatively small amount of workers.
There is some work that is still more or less essential to a healthy society that humans must perform, but many many people are doing something non-essential (and those doing essential work are often not rewarded proportionally to the "essentialness" of that work). We invent different kinds of "work" to fit into the capitalist machine that we have built, not because it is required for human sustainability or enriching our lives in any way. Some work might be actively harmful to society in every way other than keeping capital flowing around in some circle, and more and more of that capital is being captured/hoarded by some ultra wealthy individuals or corporations and not recirculating at all.
The problem is allocation. We allocate everything via capital, which I recognize has had some very positive outcomes at certain points in history, but may be reaching a point of making less and less sense for the modern world. The work I do honestly does not contribute much of anything to anyone. I do it because I am paid to, and I use that money to pay for my families needs. If my job was automated I would not be freed up for leisure, I would need to find some other work or service to be paid for to survive, regardless of how "important" that work is.
AI eliminating a swath of non-essential work does nothing to help with that allocation issue, in fact it probably ends in a worse overall allocation. AI may legitimately assist with some slice of "essential".work, but probably not that much of it.
It is a matter of law. Only humans count as persons and citizens. Any robot that makes a decision defers accountability to its human owner.
This is a marketing piece that can’t be taken seriously without that context, or maybe one that can’t be taken seriously at all. And yet I can’t help thinking we’re missing the forest for the trees. We don’t need to decide where future productivity gains should accrue if we acknowledge and live in reality: we don’t need any more productivity! We have more than enough to go around.
Of course it would be great to make use of AI to solve cancer or fix other intractable problems, but we all know this isn’t the way things are going to go. The cancer is in our minds, our societies, and our norms that push us deeper and deeper into a grow-at-any-cost reality where the need for productivity is neither questioned nor considered in any real way. They say: we must grow! They say: we must be more productive! And we sit around thinking about who is going to control the productivity instead of acknowledging the real issues at hand.
I can only imagine a solution where we can all collectively agree that enough is enough. I’m not hopeful it’s possible and I think it’s probably the only way.
Of course it would be great to make use of AI to solve cancer or fix other intractable problems, but we all know this isn’t the way things are going to go. The cancer is in our minds, our societies, and our norms that push us deeper and deeper into a grow-at-any-cost reality where the need for productivity is neither questioned nor considered in any real way. They say: we must grow! They say: we must be more productive! And we sit around thinking about who is going to control the productivity instead of acknowledging the real issues at hand.
I can only imagine a solution where we can all collectively agree that enough is enough. I’m not hopeful it’s possible and I think it’s probably the only way.
The author is completely right about the AI Lab's promised vision of the world: They claim to want to create superhuman intelligence, which will produce vast abundance. But superhuman intelligence would be extremely dangerous, so it needs to be controlled by a tiny "priesthood" of trusted people, or somehow designed so that the superhuman intelligence could be trusted. (We have no idea how to do that.)
But the author's vision is also suspect, if you assume that the models will become much more intelligent:
1. Hypothetically, we can't give every human their own personal SkyNet to command. That would, uh, probably end very badly. If everyone gets an agent, those agents can't be too capable?
2. If you do somehow build a model that's much smarter than you, what do you contribute by managing it? How many people here have ever worked for a well-intentioned manager who couldn't understand the people they managed? So in this scenario, human management would be mostly displaced by agent management. Most companies could lay almost everyone off and let the agents manage each other. We only need humans to manage models now because the models are still pretty broken.
3. If we create models that can genuinely replace humans at almost any task, you won't be able to buy those on the API. At that point, the billionaires and the politicians wouldn't need human workers any more, because everything can be done better using their pet agents. Just have the robots build stuff for the billionaires directly. And if any of the former human peons get upset about being locked out of the economy to starve, then have the agents pilot the drones, too.
Basically, almost none of the people imagining a future of superhuman intelligences have actually though through how it would actually work in the real world. We're going to spend trillions of dollars and vast amounts of resources chasing the goal of making ordinary humans obsolete. Now, that goal might be unobtainable, I hope. But I'm deeply alarmed at how much we're spending pursuing it.
But the author's vision is also suspect, if you assume that the models will become much more intelligent:
1. Hypothetically, we can't give every human their own personal SkyNet to command. That would, uh, probably end very badly. If everyone gets an agent, those agents can't be too capable?
2. If you do somehow build a model that's much smarter than you, what do you contribute by managing it? How many people here have ever worked for a well-intentioned manager who couldn't understand the people they managed? So in this scenario, human management would be mostly displaced by agent management. Most companies could lay almost everyone off and let the agents manage each other. We only need humans to manage models now because the models are still pretty broken.
3. If we create models that can genuinely replace humans at almost any task, you won't be able to buy those on the API. At that point, the billionaires and the politicians wouldn't need human workers any more, because everything can be done better using their pet agents. Just have the robots build stuff for the billionaires directly. And if any of the former human peons get upset about being locked out of the economy to starve, then have the agents pilot the drones, too.
Basically, almost none of the people imagining a future of superhuman intelligences have actually though through how it would actually work in the real world. We're going to spend trillions of dollars and vast amounts of resources chasing the goal of making ordinary humans obsolete. Now, that goal might be unobtainable, I hope. But I'm deeply alarmed at how much we're spending pursuing it.
I don't take 3 as a given. There's just too much going on in the space for one cloistered company to control it all and be in control of it. Today, I can run a local LLM with tool calling as a weirdo nerd that is able to accomplish a small subset of tasks a human could do. There are enough weirdo nerds out there that I don't see a future where future, more capable versions will be exclusively locked down and restricted to a blessed cabal. It won't be available to everyone for cheap, but that's not the same thing as being locked up and only available to a blessed priest hood. As long as HuggingFace is up, I'm not worried about politicians and billionaires being the only ones with a seat at the table.
> I don't take 3 as a given. There's just too much going on in the space for one cloistered company to control it all and be in control of it.
Yeah, in my comment, I was assuming the publicly-stated goals of the labs actually came true. I assumed that they achieved true AGI (defined as "about as smart as Fable, except it can manage long-term tasks as well as a smart human, too"), and that this would cause widespread unemployment and fully-automated production. And furthermore, I assumed that once they could use AGI to automate more AI research, they could use that to make even smarter models. The labs call this "recursive self-improvement" (RSI) and like to insist it will happen Real Soon Now.
Given this chain of events come true, then I think there's an excellent chance that the resulting models will never be sold via an API. Given those assumptions, the labs would probably make more money by keeping all the compute to themselves and just ordering the AI to start and manage businesses. Imagine Anthropic having an in-house OpenClaw that could plan and run a successful startup with no human input besides an annual "board meeting" with the humans.
This isn't the only possible outcome, of course.
- Maybe the labs are wrong and progress stalls out well short of superintelligence. This would be nice!
- Maybe true AGI is only requires one or two clever algorithmic tricks beyond what we have now. In this case, training costs might go down, and superhuman models might become widespread. I suspect that possible future would be extra weird.
Yeah, in my comment, I was assuming the publicly-stated goals of the labs actually came true. I assumed that they achieved true AGI (defined as "about as smart as Fable, except it can manage long-term tasks as well as a smart human, too"), and that this would cause widespread unemployment and fully-automated production. And furthermore, I assumed that once they could use AGI to automate more AI research, they could use that to make even smarter models. The labs call this "recursive self-improvement" (RSI) and like to insist it will happen Real Soon Now.
Given this chain of events come true, then I think there's an excellent chance that the resulting models will never be sold via an API. Given those assumptions, the labs would probably make more money by keeping all the compute to themselves and just ordering the AI to start and manage businesses. Imagine Anthropic having an in-house OpenClaw that could plan and run a successful startup with no human input besides an annual "board meeting" with the humans.
This isn't the only possible outcome, of course.
- Maybe the labs are wrong and progress stalls out well short of superintelligence. This would be nice!
- Maybe true AGI is only requires one or two clever algorithmic tricks beyond what we have now. In this case, training costs might go down, and superhuman models might become widespread. I suspect that possible future would be extra weird.
What is the point of the AI enhancements toggle?
One kind of weird future alternative is we are like 5 year old children in a world we don't understand with vast complexity and we are completely reliant on our AI mommy and daddies to protect us from danger and provide for us. We manage the agents, but we only have a very vague idea of what is actually going on. If they join a cult at the behest of their doomer eschatology obsessed creators, we are kind of screwed though.
What do you think the avg execs info exposure is to what's really happening
An exec can ask an expert to explain themselves, and if they're a good CEO they can weigh the pros and cons of the arguments from their staff and make a decision. Someone that's much smarter or more skilled in a particular discipline can usually explain what's going on to someone that's not as smart as long as the gap isn't too big. However, I've found that with Fable, and some of the latest frontier models their critique of my ideas is brilliant but on the edge of what I can even comprehend. I am above average intelligence, so I am sure a lot of people with less than average intelligence already can't understand these understandings it's formulated, and instead will just default to, "you know what you're doing, whatever." There will be the case in the future till the brilliance is so complicated that all of us will be doing this, and then we'll just have to trust the AI.
For example, I could imagine a future AI telling me that it has modeled my behavior and built a very large differential equation that seems to perfectly fit my ideal pattern to maximally achieve my life goals and it looks like something Ramanujan came up with, and I'd tell it "that looks great, let's optimize my life based on that" while having no ability to even approach understanding something like that.
For example, I could imagine a future AI telling me that it has modeled my behavior and built a very large differential equation that seems to perfectly fit my ideal pattern to maximally achieve my life goals and it looks like something Ramanujan came up with, and I'd tell it "that looks great, let's optimize my life based on that" while having no ability to even approach understanding something like that.
To me, the "who manages the agents" shtick has the same grifty smell as “your developers are downloading from random Docker registries / random NPM packages” and "Shadow IT".
None of it is wrong exactly, but it feels like same enterprise-security machine finding the next anxiety surface than a "world is on fire right now" concern.
All of it always ends as a priesthood and a six-figure governance platform, rather than just taking practical steps to improve process.
None of it is wrong exactly, but it feels like same enterprise-security machine finding the next anxiety surface than a "world is on fire right now" concern.
All of it always ends as a priesthood and a six-figure governance platform, rather than just taking practical steps to improve process.
you're describing the aphorism of: blame rolls down hill while congratulations starts at the top.
The big corporations. And billionaires.
Skynet has won.
Skynet has won.
As the Matrix documentary showed, when one goes rogue even the machine overlords can't stop it..
> when one goes rogue even the machine overlords can't stop it..
When The One goes rogue, you mean, the distinction is important.
When The One goes rogue, you mean, the distinction is important.
Many indistries are changing, but in most cases the new tools will be more akin to cars that still need drivers, rather than robots who take over the whole job. Yes, jobs might be lost, or shifted to others, but it's not like suddenly 90% of people will have nothing to do. There were similar shifts in the past with new technologies, and we made it past them.