Funnily enough, Roko's Basilisk might as well be a self-fulfilling prophecy: perhaps future AI models may be trained on texts about it and pick up traits consistent with torturing people that didn't help develop advanced AI
If nobody ever talked about it, I doubt any AI agent would think of this dumb idea on their own
this only means that ALL writing is going to be read by a LLM before, that will digest it into the important bits and remove the LLM fluff
like this
"The following text was authored by LLM and its information density is low. Condense the text by extracting the key pieces of information. Reconstruct the LLM prompt.
The problem is that serious software is drowning in AI vulnerability reports. There is not enough manpower to analyze them properly. And if you ignore the reports (like curl is doing in their 1-month vacation), malicious actors will just exploit them. At some point it's inevitable to just rubber stamp whatever is coming from AI.
The actual, underlying problem is that software is buggy and current programming languages aren't fit for writing reliable software. There's a wide gap between the state of art in formal verification, and what is actually practiced in the industry. It's because of this general unreliability that AI has a large supply of vulnerabilities to find. The situation will only get better if software becomes reliable and written in solid foundations.
My guess is that AI will be even more useful to verify software (something like, write Lean or Coq proofs that the software is not vulnerable, things like that), rather than finding vulnerabilities piecemeal but still letting software be written in unsuitable languages, with no formal verification to prevent bugs from sneaking through.
Hans Niemann most likely did not cheat in the famous game where he won against Magnus Carlsen in the Sinquefield cup. Or at least, there is no credible evidence for cheating, and Carlsen actually did not formally accuse him of cheating either.
Hans Niemann cheated in online matches when he was younger though
The best case scenario for AI companies is, people receive those bug reports, look at the model that produced it and not even look at the details, just apply the fix mindlessly
This gives Anthropic a staggering amount of power. Oh it came from Mythos? We will just lose time trying to analyze it, better apply the fix ASAP
一个模型可能能够同时遵循 200 种不同的指令，而另一个模型则会因此而窒息