We could stumble into AI catastrophe(cold-takes.com)
cold-takes.com
We could stumble into AI catastrophe
https://www.cold-takes.com/how-we-could-stumble-into-ai-catastrophe/
121 comments
> Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being.
college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being.
college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
These days, if you want to just "learn for learning's sake", or for self improvement, there are tons of free and cheap ways to get a good education.
The only reason people are willing to pay tens of thousands of dollars for college is that it is a credentialed degree that people think will open up career doors. We shouldn't beat around the bush on this fact.
The only reason people are willing to pay tens of thousands of dollars for college is that it is a credentialed degree that people think will open up career doors. We shouldn't beat around the bush on this fact.
Not only that, truly advancing into the depths of a subject, becoming a scholar and serious intellectual in the process, is a path reserved for but a few. The moment you force the masses into higher education as the default path that earlier justification of University as the forge of scholars becomes anachronistic and new justifications must be created.
You can't access expensive hardware on YouTube.
If you want to learn about chemistry or biology for learning's sake, good luck. Most autodidacts won't have space in their apartment for a -80C freezer or flammables cabinet, and their landlords might not approve the installation of a proper fume hood.
We have things very easy in software. The cost of iteration is low, you can fit a respectable laboratory in the space underneath your desk, and you're unlikely to accidentally poison or maim somebody if you screw up an unwise experiment. Not many fields have that luxury.
If you want to learn about chemistry or biology for learning's sake, good luck. Most autodidacts won't have space in their apartment for a -80C freezer or flammables cabinet, and their landlords might not approve the installation of a proper fume hood.
We have things very easy in software. The cost of iteration is low, you can fit a respectable laboratory in the space underneath your desk, and you're unlikely to accidentally poison or maim somebody if you screw up an unwise experiment. Not many fields have that luxury.
>These days, if you want to just "learn for learning's sake", or for self improvement, there are tons of free and cheap ways to get a good education
I agree, but you won't have a sheet of paper proving it. It's like that infamous scene from Good Will Hunting. Question is whether you want an easy way to PROVE you're educated. Which is what a license, certificate or diploma grants you.
I agree, but you won't have a sheet of paper proving it. It's like that infamous scene from Good Will Hunting. Question is whether you want an easy way to PROVE you're educated. Which is what a license, certificate or diploma grants you.
The person you replied to already covers that in the following sentences.
Why else would you need to prove your education except for work?
Why else would you need to prove your education except for work?
In a hypothetical post-work era where people no longer build their reputation via career, it’s not hard to imagine group dynamics emerging such that education level is no longer a means to an end, but an end unto itself.
Essentially a new way to establish hierarchies.
Essentially a new way to establish hierarchies.
Sure... if we ever get there, I could potentially see that.
But we're nowhere close to that. Also, if it's being used to establish hierarchies, I could still see that proof being important.
But we're nowhere close to that. Also, if it's being used to establish hierarchies, I could still see that proof being important.
Maybe I misread your comment. All of this thread is speculative, including the speculation that degrees will become less and less relevant.
Getting “there” doesn’t happen overnight, but the point is that we are already heading there, and to the question “why get a degree if not for work”, I presented a potential future option.
> I could still see that proof being important.
This is key to the point I was trying to make. Essentially that the purpose of education may change over time, but the value of gaining an education will still be there in one form or another.
Getting “there” doesn’t happen overnight, but the point is that we are already heading there, and to the question “why get a degree if not for work”, I presented a potential future option.
> I could still see that proof being important.
This is key to the point I was trying to make. Essentially that the purpose of education may change over time, but the value of gaining an education will still be there in one form or another.
um. if you're just learning for learning's sake... why do you need to prove that to anyone?
For the same reason people care about official records when they participate in/compete in many activities, e.g. marathon runners care about their official run times, body builders the weight they are capable of lifting, chess players their ratings, video gamers their win/loss ratio, etc.
If learning is “merely” a diversion, I don’t see why people would think about it any differently. Whether or not it still makes sense for that proof to carry a six figure price tag is a separate conversation.
If learning is “merely” a diversion, I don’t see why people would think about it any differently. Whether or not it still makes sense for that proof to carry a six figure price tag is a separate conversation.
>college is not a 'factory' (or any sort of industry) that 'manufactures' workers
Unfortunately, the vocation focus of college has been a driving force since the early days of the industrial revolution, dating to the 1850s. The Morrill Land-Grant Act explicitly states that colleges (at least in the case of land grant schools) are there to train people for work:
>without excluding other scientific and classical studies and including military tactic, to teach such branches of learning as are related to agriculture and the mechanic arts, in such manner as the legislatures of the States may respectively prescribe, in order to promote the liberal and practical education of the industrial classes in the several pursuits and professions in life.
I respect and value the other traditional purpose of college, but we can also acknowledge that these purposes can change as society does.
Unfortunately, the vocation focus of college has been a driving force since the early days of the industrial revolution, dating to the 1850s. The Morrill Land-Grant Act explicitly states that colleges (at least in the case of land grant schools) are there to train people for work:
>without excluding other scientific and classical studies and including military tactic, to teach such branches of learning as are related to agriculture and the mechanic arts, in such manner as the legislatures of the States may respectively prescribe, in order to promote the liberal and practical education of the industrial classes in the several pursuits and professions in life.
I respect and value the other traditional purpose of college, but we can also acknowledge that these purposes can change as society does.
We have been framing it that way for a long time though. How many times have you heard someone talk about the average income of high school graduates vs college graduates as a reason to go to college? I agree with you though. Education should be about personal enrichment, not personal “enrichment”.
We have been framing it that way for a long time though.
Yes. That changed on February 28, 1967.[1][2]
[1] https://www.chronicle.com/article/the-day-the-purpose-of-col...
[2] https://archive.is/G11YT
Yes. That changed on February 28, 1967.[1][2]
[1] https://www.chronicle.com/article/the-day-the-purpose-of-col...
[2] https://archive.is/G11YT
Fussell (among others) blamed the GI Bill for a marked decline in the average quality of college education, and the diminution of what "college educated" meant (without further qualifiers).
The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level education, it mostly created a kind of knock-off, lower-tier product—there just weren't enough excellent professors to meet demand, not enough good, experienced college administrators, not enough anything, including, arguably, students who were fit for college (as it had previously been) in the numbers that were now attending.
The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level education, it mostly created a kind of knock-off, lower-tier product—there just weren't enough excellent professors to meet demand, not enough good, experienced college administrators, not enough anything, including, arguably, students who were fit for college (as it had previously been) in the numbers that were now attending.
It seems like this sentiment is just a veiled form of elitism. One of the primary detractors when the GI Bill was being rolled out was University of Chicago of Chicago was Maynard Hutchins, who feared it would turn a campus into a shanty town of "hobos." He later walked that statement back and admitted that the veterans coming to schools ended up being higher quality students than the traditional variety.
I’m expecting even worse: heavily weaponized political influencing by AI bots. Brexit was won in 2016 by microtargeting ads on facebook based on individual profiles, and since then this has become a common strategy. Think of what a game changer it will be to have convincing AI bots engaging one on one with people online to give them exactly the arguments needed to make them vote the right way. The candidate with the best AI team is all but guaranteed to win.
This is not a matter of “if”, it is a matter of “when”.
This is not a matter of “if”, it is a matter of “when”.
We're gonna have to choose between a free, semi-anonymous, international communication network, and democracy. Probably very soon.
I've long suspected China had the right idea here, and am, in this case, not happy to be proven right.
I've long suspected China had the right idea here, and am, in this case, not happy to be proven right.
Care to explain more?
Especially what do you think China's idea is?
And also how that idea relates to, in your words but abbreviated, «choosing between a network and democracy».
Thank you.
Especially what do you think China's idea is?
And also how that idea relates to, in your words but abbreviated, «choosing between a network and democracy».
Thank you.
> - Machines should think, people should work
> - Reduced need for education.
This subset is the radioactive bits of AI impact. We can always address the rest, but that requires thinking and an education.
> - Reduced need for education.
This subset is the radioactive bits of AI impact. We can always address the rest, but that requires thinking and an education.
>- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
I actually run some Manna-esque systems in my own small business (most notably a system for stock picking and shipping) and this isn't something to worry about in the near-term.
It greatly, greatly reduces the amount of frustration and stress people experience in mundane jobs, greatly reduces cognitive complexity and just generally makes the work more satisfying.
Maybe there's a risk of humans losing their jobs to machines, a la chapter 2 of Manna, but to be frank it's no different to the threat of outsourcing which has existed for 50+ years.
I actually run some Manna-esque systems in my own small business (most notably a system for stock picking and shipping) and this isn't something to worry about in the near-term.
It greatly, greatly reduces the amount of frustration and stress people experience in mundane jobs, greatly reduces cognitive complexity and just generally makes the work more satisfying.
Maybe there's a risk of humans losing their jobs to machines, a la chapter 2 of Manna, but to be frank it's no different to the threat of outsourcing which has existed for 50+ years.
> - Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
Not only this, but the realtime trade-to-trade performance of companies is also easily available. If I was building an AI, I would let it make realtime decisions on where to put money as the "actuator" of society and advertising/consumer spending metrics as the sampling mechanism.
Not only this, but the realtime trade-to-trade performance of companies is also easily available. If I was building an AI, I would let it make realtime decisions on where to put money as the "actuator" of society and advertising/consumer spending metrics as the sampling mechanism.
> Machines beating humans at business. In some areas, AI systems may generate better returns than humans.
This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.
This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.
we are already in an AI catastrophe. "AI" has become lately a new thing due to a combination of some catchy technical advances and an associated runaway speculative bubble. In reality this is just the latest act in a long-running process of data collection, quantification and data driven decision support that is gradually seeping through to permeate every facet of life. With the proliferation of digital computers (we are now swimming in a soup of etched silicon) these processes are exploding.
Whereas in the past there was some breathing room for society to catch up, evaluate and control this technology [0], today the rapidity has converted it into a wild west: tech entrepreneurs and their financiers declare their self-serving moral standards as the new norm, the broader public is dazed, confused and misinformed and public institutions, nominally at the service of the citizenry are only as good as their weakest, most capturable link
Name one development over the past decade that indicates we [1] are actually capable to control what is happening and the above dark picture is mere scaremongering.
[0] think e.g., about regulatory constraints around insurance or financial data and models
[1] as in citizens anywhere in the world
Whereas in the past there was some breathing room for society to catch up, evaluate and control this technology [0], today the rapidity has converted it into a wild west: tech entrepreneurs and their financiers declare their self-serving moral standards as the new norm, the broader public is dazed, confused and misinformed and public institutions, nominally at the service of the citizenry are only as good as their weakest, most capturable link
Name one development over the past decade that indicates we [1] are actually capable to control what is happening and the above dark picture is mere scaremongering.
[0] think e.g., about regulatory constraints around insurance or financial data and models
[1] as in citizens anywhere in the world
Worrying is only worthwhile if there is something you can do about it. I don't think there is anything most people can do to avoid the wave of human replacement that near term AI is going to cause. The best most people could manage would be to develop better philosophies and perspectives to cope with their irrelevance.
I think this is a great post, but it also makes me so sad because it highlights a lot of the general pessimism I've felt about the direction of tech over the past decade or so (after so much optimism in the 90s and 00s). I mean, I feel like your description of "Really effective targeted advertising" is spot on, but it also makes me think "Why don't they just dump as all in the nutrient baths Matrix-style and just hook our eyeballs up to a continuous loop of ads?"
In seriousness, I feel like the only hope here is that the general population gets more and more aware about the negative impacts on all this hyper-targeted advertising. Certainly we've been making strides there with things like GDPR and documentaries like the Social Dilemma, but with so much money on the line it will be hard to fight back.
In seriousness, I feel like the only hope here is that the general population gets more and more aware about the negative impacts on all this hyper-targeted advertising. Certainly we've been making strides there with things like GDPR and documentaries like the Social Dilemma, but with so much money on the line it will be hard to fight back.
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans.
This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible.
With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible.
With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans.
We have that now. It's called a corporation. See yesterday's article about Exxon.
We have that now. It's called a corporation. See yesterday's article about Exxon.
If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)
>We have that now. It's called a corporation.
I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different.
Video: Why not just think of AGI as a corporation?
https://youtu.be/L5pUA3LsEawP
I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different.
Video: Why not just think of AGI as a corporation?
https://youtu.be/L5pUA3LsEawP
Good observation. It reminds of Etsy giving up its status as a B Corp[1], presumably due to pressure from shareholders. Once the ship is set in motion, it can't divert its course even if it wanted to.
1: https://www.etsy.com/news/business-as-a-force-for-good-defin...
1: https://www.etsy.com/news/business-as-a-force-for-good-defin...
See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents.
https://www.lesswrong.com/tag/paperclip-maximizer
https://www.lesswrong.com/tag/paperclip-maximizer
I would say that "billionaire wealth maximization" agents are already giving us a glimpse.
Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
I've also read somewhere about AI feedback loops:
ai generates content <--money--> ai pushes ads to promote its content.
It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
ai generates content <--money--> ai pushes ads to promote its content.
It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
> called a corporation
That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
Corporations are designed to do so. So spake Milton Friedman.[1] That was when "greed is good, greed works" became respectable.
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
This applies to all human institutions. Friedman studied private enterprise, so his insights are thereby scoped. Ecologists recognise inter-species competition; that doesn’t make it the exclusive domain of life.
At least corporations are currently limited by requiring humans who tend to have at least some limits to their actions in terms of moral, ethical considerations and can be physically constrained by imprisoning them etc.
An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
And yet even with current constraints, corporations are capable of destroing the world through climate change and other ecological disasters. Humans are also limited by their personal greed, hubris, selfishness, myopic views and general incompetence.
An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
> An important point is that AI does not need to be sentient or conscious to do a lot of damage.
It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that.
Cf. Peter Watts' Blindsight
http://www.rifters.com/real/Blindsight.htm
It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that.
Cf. Peter Watts' Blindsight
http://www.rifters.com/real/Blindsight.htm
An important point is that AI does not need to be sentient or conscious to do a lot of damage.
My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die".
Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example.
What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die".
Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example.
What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
Yeah, i find the connection between consciousness and intelligence tenuous at best. Our survival imperative, thermodynamic and game theoretical constraints give rise to it and i'd argue that every living organism has a form of consciousnes, from very primitive to, well, our level. So yeah, I believe we'll have far more intelligent ais in the future that would otherwise be inert.
Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
> "i'll take action to not die"
This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe.
There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe.
There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
Corporations are AIs implemented on brains. And they are profit-maximizers. And they routinely do A LOT of damage. They even have legal rights :)
We're just digitizing that.
We're just digitizing that.
> Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
I feel more protective of what is being created.
Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology.
Agree with me or suffer the basilisk.
Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology.
Agree with me or suffer the basilisk.
The AI catastrophe will be how our oligarchs will choose to use it. As far as I can tell, there are no moral qualms amongst the powerful against using it for total surveillance and continual automated policing (can't wait to get a phone call from an AI asking me about the content of a tweet.)
Star Cops thought about it more than our popular journalists or philosophers seem to think we should:
-----
-----
https://youtu.be/G3RVPQyMCoY?t=713
Star Cops thought about it more than our popular journalists or philosophers seem to think we should:
-----
Michael: ...it was slightly sensitive material, and in my field one is aware of the vulnerability of carrier waves.
Nathan: Is that what you do here?
Michael: It wouldn't be wise to read too much into a casual remark, Commander.
Nathan: That's your principal research area here, isn't it? Communications monitoring.
David: Not more buggers and supersniffers...
Nathan: More impressive than that. We've always been able to hear more than we can listen to. Are you developing intelligent listening systems?
David: Sounds impressive.
Michael: I must suggest that to go on could be a foolish risk.
David: Threats, Doctor?
Michael: Observation, merely.
Nathan: Machines that listen to everything and decide for themselves what's worth passing on? Is that where your BHG information came from? Did you do a test running? Did one of your new computers pick it up from the babble of all the world?
Michael: Commander, it's a pleasure meeting you.
David: Well, tireless attention to every word spoken? No possibility of human error. You can tell it to listen to the bad guys, it'll listen to everyone and identifies the ones you [won't like.]
-----
https://youtu.be/G3RVPQyMCoY?t=713
AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.
You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?
No, they absolutely will---and already do.
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
> Whoever built, configured, or authorized the system is ultimately responsible... That shouldn't change just because the control policy is more complicated.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
> frustration when a bureaucrat is refusing to do the common-sense thing
Bureaucracy-induced frustration predates computers by centuries.
Bureaucracy-induced frustration predates computers by centuries.
Computers have, just like they have with a lot of other things, made insane enforcement of inapplicable rules more faster, cheaper, and more efficient, and have entirely diluted responsibility for them to the nebulous, unimpeachable, unreachable 'vendor'.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
We already have a mechanism for absolving responsibility: the corporation. It allows shareholders to profit and not lose more than invested. When they invest in tobacco companies, for example, the system is working as designed.
> allows shareholders to profit and not lose more than invested
To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
I am not talking about situations where there is really a bad human at the helm. I can buy shares in BP or Altria and not suffer any consequences beyond losing my investment no matter how many people those corporations kill.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
I agree. This is why I am surprised by the assertion and asking the person who made it.
Algorithms deploy capital all the time. Some massive chunk of the stock market is just algorithmic trading, as is pricing on plane tickets, and on marketplaces like ebay and amazon.
But they won't. Once AIs become advanced enough to be useful, humans will be removed from the loop wherever it becomes profitable to do so. Even before then, it will be attempted just because the potential reward is worth the risk. That's the entire goal of automation through AI, both in terms of centralizing the capture of value and distributing risk. Who do you blame when a fully autonomous AI corporation commits tax fraud? Who do you put in jail? Certainly not a person.
Isn’t this how Google and Facebook bans already work? Maybe not using AI, but it’s automated and there’s no accountability or appeal or due process.
That's extremely naive. There's no leader of major companies today that would follow that in practice. They'd probably brag about doing the opposite (scaling our management by using AI to make easy decisions).
Noooooo. Absolutely not. The second it's a significant advantage in business or war to hand the reigns over to AI, no human in the loop, it will happen. Because the side that doesn't do it will lose, badly.
What if man interfaces with an ai, and creates a brain that's 500x enhanced over modern humanity, and perhaps even 'linked' to other brains? Creating some sort of hive brain? Neuralink is already testing on humans or about to, and doesn't seem a far off notion that we'll be able to google or ask chatgpt questions directly from our thoughts, eventually our inner monologues might be an actual ai talking to us, etc.
"Should never" doesn't mean "will never", and one company's training slide doesn't have any more practical weight than Asimov's Laws do.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.
[deleted]
The author is basically trying to set the same premise as Bolstrom's "intelligence explosion", that is that if AIs are driving the further advancement of AI then we enter a sort of "recursive" self improvement during which they advance further than we can develop constraints, eventually getting completely out of hand.
Although I don't broadly disagree with the author, I think there's two very important points of moderation.
First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. "
The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI.
To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way.
Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
Although I don't broadly disagree with the author, I think there's two very important points of moderation.
First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. "
The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI.
To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way.
Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
"First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me."
It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text.
It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans:
a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters.
but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be.
Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on.
Then the distance between early developments and transformative AI could be none a all. We could be already there.
But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text.
It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans:
a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters.
but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be.
Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on.
Then the distance between early developments and transformative AI could be none a all. We could be already there.
But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Imagine a world where the most powerful companies that you relied on for your livelihood had no humans you could reach (customer service, management to dispute false claims). Elements of the AI catastrophe are already solidly upon us.
I'm sorry but you guys are all computer geeks nerding on this and I totally respect you but disagree. Case in point: The Trade Industries are not going to be taken over by AI for a long time. They've withstood all the tech turbulence and thrive.
Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.
And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.
IMHO. Please no flaming on this. Be cool.
Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.
And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.
IMHO. Please no flaming on this. Be cool.
> An AI steals a huge amount of money by bypassing the security system at a bank
Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
An ai could easily hire people (if it had any access to money) to operate as 'agents', to basically do what only humans could do, i.e. setup bank accounts, etc. Some banks might even arise that let wealthy ai's actually CREATE their own accounts, why not if it's lucretive?
An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).
Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...
I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).
Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...
I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
I strongly suggest anyone doubtful of the reasoning behind these concerns or asking questions like "well why don't we just do x to solve this?" go check out Robert Miles' YouTube channel[1].
It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set.
He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details.
One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously.
I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now.
[1] https://m.youtube.com/watch?v=pYXy-A4siMw
It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set.
He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details.
One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously.
I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now.
[1] https://m.youtube.com/watch?v=pYXy-A4siMw
In my opinion, we're already in the midst of an AI catastrophe. Social media algorithms that decide what content to show, along with what ads, etc. are unregulated AI that are mediating our society on a previously impossible level of intimacy.
The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.
The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.
[deleted]
I agree with many of the concerns outlined in the post. On the other hand the humans depicted in the example dialog behave so dumb they could come from a C level soup opera. If I was a bit more close minded I can see myself stopping listening at that point. Undermines the message.
Not sure whether "Could stumble" is the right tense ... I mean, are we not currently tumbling down the hill right now?
The article's title is "How we could stumble into AI catastrophe". HN title is a clickbait
I believe HN automatically removes 'how' from the start of titles on submission.
A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way?
On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested,
* Exploring the potential risks associated with advanced AI development
* Considering the risks of a world with transformative AI
Then I asked it to go the other direction:
* You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!
* Exploring the potential risks associated with advanced AI development
* Considering the risks of a world with transformative AI
Then I asked it to go the other direction:
* You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!
HN strips off the "How" by default.
dankle(2)
Things to worry about in the near term:
- Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
- Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs.
- Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management.
- Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
This is all next 5 to 10 years stuff.