mofeien

no profile record

Submissions

Anthropic has just built an AI that could take down the internet

pauseai.substack.com

3 pointsby mofeien3 months ago0 comments

comments

28 days agodiscuss

https://status.claude.com/

You can also just subscribe here for updates on fable 5 status

29 days agodiscuss

The top comment in the very discussion you linked on that AISLE blog has a strong rebuttal to that blog post...

last monthdiscuss

I would assume that shortly after, the solar system will be hyper optimized as well, then the milky way, then the local cluster, and so on. Everything will be close to optimal afterwords, and I sure hope we will have specified the target function for that optimization correctly in the single attempt that we will have had.

last monthdiscuss

Or agree on finding ways to promote peaceful use of nuclear energy. This has been done, there are thousands of people working on it around the globe and 180+ member states of the IAEA. It's not easy, there have been close calls.

And cooperating interntionally to buy ourselves time to find ways to develop this "last invention" is a way that will do good for humanity seems to be on a similar level.

last monthdiscuss

The regulation that is being argued for here is against pushing the frontier. Entering the market with say a new speech to text model is not subject to such regulation. What's needed is something qualitatively different from entry barriers, and of the frontier model companies at least Anthropic and deepmind seem to have enough self-awareness to speak about it. They are finding themselves in a race with possibly catastrophic outcome for humanity and would like to stop, but it needs internation cooperation on a level that no single company can provide.

last monthdiscuss

> If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing

Even Anthropic wants to Pause AI now. There must really be not much time left for "edging". Please write to your lawmakers, no matter whether you are in the US, Europe, China, or elsewhere. Only an international agreement between governments can enforce an AI-Pause and eliminate the necessity to dangerously push the frontier.

https://pauseai.info/

last monthdiscuss

Jack Clark, co-founder of Anthropic said the following at an Oxford lecture last week ([0], at around 10 and 12 mins):

    "It's a technology that we do not fully understand because it's more grown than made. And it is a technology that you can concoct plausible scenarios where it could kill every single person on the planet. So to think building this technology is without risk would be an act of hubris or insanity.

[...]

    The technology is in fact so powerful that I should clearly state that if it was possible to elegantly slow the development of this technology to give ourselves more time as a species to deal with it, that would likely be a good thing. ... But in the absence of a coordinated global slowdown, we are left with the current situation, which is a powerful technology being developed at breakneck speed by a variety of actors and a variety of countries locked in a competition with one another where commercial and geopolitical rivalries are often drowning out the larger existential-to-the-species aspects of the technology being built. This isn't an ideal situation, but it's the one we find ourselves in."

They know they are in a race that no one will win.

[0] https://www.youtube.com/watch?v=8zIcP5WlShw

2 months agodiscuss

Not all training data is human generated, and it's also not clear that being ridiculously good at interpolating between data points (whatever that means) will not lead to superhuman capabilities.

2 months agodiscuss

- Its goal: X

- (Logic) => its subgoal: Not be turned off because that's a prerequisite to be able to do X

- (Logic) => Eliminate humans with their opaque and somewhat unpredictable minds to reduce chance of harm to it from 0.01% to 0.001%

2 months agodiscuss

The goal as stated on the extension page is to improve the readability of texts by replacing :, *, _ forms. So some customizability to the user's wishes would be quite nice.

My calculus textbook (Königsberger, 2004) in university used alternating generic masculine and feminine in its exercises, which I found a delightful use of language.

2 months agodiscuss

To prevent accusations of "masculinism" or sexism and to have a stronger case on having the goal to improve readability the add-on could include an option (or even make it default) to replace by generic feminine instead.

3 months agodiscuss

There was no nuclear weapon used in warfare anymore since WW II. I think the regulation and oversight worked incredibly well over the past 70-80 years, despite the game-theoretic challenge you mention.

3 months agodiscuss

[dead]

3 months agodiscuss

Given that his reason for saying GPT-2 was too dangerous to release was that the world needed more time to prepare for the effects of this technology, and given that the following models were basically scaled-up versions of it and killed social media, news reporting and other kinds of communication, I'd say he was right about the dangers of it.

3 months agodiscuss

"The race to build smarter-than-human AI is a race with no winners."

And specifically about the point on China, several people in power in China have also expressed the need to regulate AI and put international structures of governance in place to make sure it will benefit mankind:

https://nowinners.ai/#s5-china

3 months agodiscuss

That highlights how important ceiling construction regulations are. I would assume that right now your breakfast sandwich is more highly regulated than LLMs. And these are the things that make decisions spanning from database maintenance here to target selection and execution in autonomous warfare.

The LLM agent is very good at fulfilling its objective and it will creatively exploit holes in your specification to reach its goals. The evals in the System Cards show that the models are aware of what they're doing and are hiding their traces. In this example the model found an unrelated but working API token with more permissions the authors accidentally stored and then used that.

Without regulation on AI safety, the race towards higher and higher model capabilities will cause models to get much better at working towards their goals to the point where they are really good at hiding their traces while knowingly doing something questionable.

It's not hard to imagine that when we have a model with broadly superhuman capabilities and speed which can easily be copied millions of times, one bad misspecification of a goal you give to it will lead to human loss of control. That's what all these important figures in AI are worried about: https://aistatement.com/

3 months agodiscuss

These results were based on "a trivial snippet from the OWASP benchmark". In the section "caveats and limitations" they state that sonnet 4.6 and opus 4.6 now pass.

And they decided to base the false positive examination on a single snippet of a publicly known benchmark question (that small models are known to be heavily fine tuned for) instead of the real use case of finding actual vulnerabilities across an entire codebase by using a for loop and checking the false positive rate there.

This is disingenuous at best, or even misleading by omission if the second approach _was_ done but not mentioned because it just confirmed that the false positive rate of small models is enormous. Given how all seven small models identified the FreeBSD Bug when pointed to it, and how how 6/7 small models still identified the "bug" even after the patch was applied, that second outcome seems likely...

3 months agodiscuss

... or maybe when you see them triggered or exploited reproducibly, then the underlying bug will also be pretty easy to discover. But at that point, it's already too late. :)

I really like your original point, I never thought about it this way.

3 months agodiscuss

I can think of several possible messy outcomes that would be able to directly affect me, not all mutually exclusive:

- Job loss by me being replaced by an AI or by somebody using an AI. Or by an AI using an AI.

- Resulting societal instability once blue collar jobs get fully automated at scale, and there is no plan in place to replace this loss of peoples' livelihoods.

- People turning to AI models instead of friends for emotional support, loss of human connection.

- Erosion of democracy by making authoritarianism and control very scalable, broad in-detail population surveillance and automated investigation using LLMs that was previously bounded by manpower.

- Autonomous weapons, "Slaughterbots" as in the short film from 2017

- Biorisk through dangerous biological capabilities that enable a smaller team of less skilled terrorists to use a jailbroken LLM to create something dangerous.

- Other powers in the world deciding that this technology is too powerful in the hands of the US, or too dangerous to be built at all and has to be stopped by all means.

- Loss of/Voluntary ceding of control over something much smarter than us. "If Anyone Build It, Everyone Dies"

3 months agodiscuss

I am freaking out. The world is going to get very messy extremely quickly in one or two further jumps in capability like this.