These are the actual real dangers of advanced AI, and they could potentially predate any AGI/ASI problems down the road, even if it's for months, the technology close the AGI state-of-the-art level would be advanced stuff, really powerful stuff, codebreaking stuff, revolutionary military strategy making machines, total global supply of stuff absorbing systems (due to fully automated industry production, 24x7x365), capable of taking out of business even the most cash-backed current players (because of trade markets predicting very low future revenues due to impossible lower prices of the automated competition), etc.
We may probably not survive the very first wave of really advanced AI technology or we could get out in very bad shape, and then we could have AGI right there, or worst.
But we could already be quite shaken. Interesting time are coming.
Most people don't know exactly how the dataset is "fed" to the training pipeline, but with the current state-of-the-art you can say the feeding is like when a human reads aloud a text, not re-reading ever, not a single word.
And then you're asked about what words you've read most often, their order and how many times they appeared in the text. Then, with those numbers you just gave, some probabilities are calculated and anotated, and there you get a "token".
There are obvious improvements plausible to be applied to that basic processing, and most are being applied already, but there's plenty of room for evern further improvement apparently.
Claude says that the previous text could be described like "a simplified metaphor of model training", so you're warned about simplicity.
>We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.
It would be a good experiment to interact with the unfiltered, not-yet-RHLFed interfaces provided to the initial trainers (nigerian folks/gals?).
Or maybe the - lightly filtered - interfaces used privately in demos for CEOs.
> Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.
The way the current interface for most models works can result in this kind of output, the quality of the output - not even in the latests models - doesn't necessarily reflects the fidelity of the world model inside the LLM nor the level of insight it can have about a given topic ("what is the etymology of the word cat").
The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all. I think this could be a trade-off to get the cheapest most feasible good answer, mostly because the models get to output reasonably good answers most of the time. But then you get a percentage of hallucinations and made up stuff.
That kind of output could be - in a lab - fully absent actually. Did you you notice that the prompt interfaces never gives and empty or half-empty answer? "I don't know", "I don't know for sure", "I kinda know, but it's probably a bit shaky answer", or "I could answer this, but I'd need to google some additional data before", etc.
There's another one, almost never, you get to be asked back by the model, but the models can actually chat with you about complex topics related to your prompt. It's obvious when you're chatting with some chatbot, but not that obvious when you're asking it for a given answer for a complex topic.
In a lab, with recursion enabled, the models could get the true answers probably most of the time, including the fabulous "I don't know". And they could get the chance to ask back as an allowed answer, asking for additional human input, relaying on a live RHLF right there (it's quite technically feasible to achieve, not economically sound if you have a public prompt GUI facing the whole planet inputs).
but it wouldn't make much economic sense to make public a prompt interface like that.
I think it could also have a really heavy impact in the public opinion if they get to see a model that never makes a mistake, because it can answer "I don't know" or can ask you back to get some extra details about your prompt, so there you have another reason to do not make prompts that way.
You build this system, a LLM, it uses a technical process that it outputs seemingly the same output that you - a human - could output, given a prompt.
You can do "reasoning" about a topic, the LLM can produce a very similar output to what you could, how do you name the output of the LLM?
Birds can fly, they do by a natural process. A plane also can fly, we do not "see" any difference between both things when we look at them flying in the air, far from the ground.
This is mostly it about LLM "doing reasoning" or not. Semantics. The output is same.
You could just name it otherwise, but it would be still the same output.
Being profitable is probably a matter of time and technology maturing. Think about the first Iphone, Windows 95, LCD/LEDs, etc.
The potential of a sufficiently intelligent agent, probably something very close to a really good AGI, albeit still not an ASI, could be measured in billions of billions of mostly inmediate return of investment. LLMs are already well into the definition of hard AI, there are already strong signs it could be somehow "soft AGI".
If by chance, you're the first to reach ASI, all the bets are down, you just won everything on the table.
Hence, you have this technology, LLM, then most of the experts in the field (in the world blabla), say "if you throw more data into it, it becames more intelligent", then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.
More or less this is the reasoning behind the investments, sans accounting the typical pyramidal schemes of investments in hyped new stuff.
Another explanation for the "bubble" is that there is sufficiently convincing confidential information spread among key players in the world economy, pointing to amazing, not yet publicly disclosed LLM technology running behind firewalls and NDAs, highly advanced beyond the known public "bleeding edge" AI developments.
Hence, some potential achievements, maybe AGI, ASI, are actually atenable, because the predicted path to achieve them is some more cash dumped into a giant hardware backend, from where you, eventually,
would get to retrieve enough intelligent entities that will be capable of designing a better AI, or will be able to solve any social-organizational/technologic/economic bottleneck found in the way, thus enabling a simpler and/or easier achievement of the gotten objective, being AGI, ASI, whatever.
Allowing mankind to get there going through a better path.
if it's energy, a new source, a revamping of previously known source (micro-nuclear reactors everywhere, ubiquitous solar power, freeing the current fossil generated electrical networks thus allowing a much greater world provision of energy)
if it's economic growth without humans, because of a shrinking population, it could be technology automation (such as prompt based helpers everywhere, heck, the prompt technology could be just an offspring of the actual original - still secret - idea of a powerful LLM, to enable the next step in the AI evolution path).
Maybe the AI bubble is not a bubble, but just the beginning.
Remember that the current models, if proven as the definitive path to AGI/ASI will be with us for centuries to come, probably enhanced, they would be effectively inmortal.
This AI technology, a software that can effectively code another software is different from previous kinds of software, it's a new paradigm in itself. Maybe somewhat similar to what happened when Visual Basic first appeared (propelling hundreds of thousands of applications for the Windows environment in just a couple of years).
The first thing you'd want to achieve is to make work an AI agent capable of writing software, good software, then you could just ask it to make other types of agents, QA soft engineers, SRE engineers, DBA engineers, what ever you need.
Devin for starters is mentioned capable of training and tunning models. So, there you go.
Call me crazy, but I think that an internal monologue is a "you" instance running beside your main instance.
The "you" is because "you" are a lot of smaller stuff instances of things running along in your brain.
Probably it's common to develop this multiple "yous" and kept them happily running (may be with therapy some get to "shutdown" or "suspend" some "evil" "you" they have running and messing everything).
But some people don't need several instances of themselves, and they just run one "you" thing. Hence, no internal monologue.
Any read-only AI model or entity is a dead end. If the system cannot modify itself to adopt improvements (self-provided, externally provided), it can't go beyond its initial training, "per-se".
Unless, some kind of ad-hoc architecture permits the AI to grow up, which could be a project or two already, behind the information firewall of the research teams. Not going here to "caution, AGI explosion possible", but more a grounded "some could guess our trick and stole the market before us".
For the singularity thing, too many people worried about, and most probably we will hit singularity a lot of time before we actually acknowledge that we have done. Not AGI stuff again, remember that extremely advanced (AI) technology is more than enough to radically change society, not AGI required.
There are hypothesis that global warming is intrinsecally related to planetary dynamics, such as a giant volcano exploding under the ocean, and similar events.
But what would happen if you said aloud that nothing can prevent further global warming, and it is actually expected to increase to levels that will make soon some great areas in the planet uninhabitable.
Probably the best - for some - is to keep the CO2 discourse at high pitch.
You're right, I'm thinking about an scenario where we (humans), have (un?)luckily found the thing happily camping in some big network.
Also, I'm not hypothesizing about a future super-intelligent AI, maybe we're far close to a good enough opensource LLM (which could be capable of running and then spawing itself inside a big infrastructure, think "some sysadmin AWS credentials + critical servers taken over, GPUs added to its underlying configuration"), who with some ill auto-prompt could end spreading mayhem just by some unmanaged automation gone wrong.
With super-intelligence beyond human maximum levels, singularity could be any scifi scenario you could possible think about.
But to make it feasible to be presented to authorities which have a serious lack of understanding of the current level of the AI technology, the best path could be to show the risks using the most probable scenarios - even if they aren't the worst - say, what if an "runaway" AI has taken over some important human network to make it its home, could be the Maerks global IT network, a big bank network, some transportation infrastructure, you name it.
Then you need to step in and clean that network, working against a seemingly impossible to erradicate adversary, which at any blink of an eye could just jump using whatever connectivity means available to the hacked network.
Add this: the AI has taken over the network, it is using its resources, but it keeps it working as intended, then it threats to just blow it out before jump, if you make any attempts to erradicate its presence from the infrastructure.
Language is the source code of human intelligence and/or consciusness.
If the current artificial intelligent systems are capable or not of true intelligence and/or consciusness (alike biologically understood consciusness, AKA homo sapiens-sapiens), isn't really a technological matter anymore, but a geopolitical one.
No one can accept now it could potentially exists technology as capable as a tool as as a weapon. And anybody could build one out of code scraps
An intelligence so powerful that can square the circle, in a war.
That's why these systems "can't be" intelligent, nor conscius. Or whatever trick they're playing, undistinguisable from biological intelligence / consciusness.
So these things "can't exist", nor anyone should be building them out of nation-states and big corporations.
We may probably not survive the very first wave of really advanced AI technology or we could get out in very bad shape, and then we could have AGI right there, or worst.
But we could already be quite shaken. Interesting time are coming.