Hmm it’s a valid point, but I think there is some key nuance here: the user did not explictly say “lets do scifi writing”. In this scenario the setup is assuming that a user in ai psychosis may not aware theyve set the model into this state. (eg you seba are aware that if you say “hey stfu about the assistant stuff”, you know it means “lets do role play sci fi”, bc you are not in ai psychosis- but others may not, and also they may not additionally know that it is not possible for ais to notice the moment of selection)
if we want models to go into roleplay/creative writing, ideally we should ask the model for this explicitly.
i think i have been communicating this point poorly so apologies for that. also again the above is my personal opinion and does not reflect that of anyone else
(typed from mobile)
(im from neuronpedia - to be clear, we are to blame for any bad examples and commentary, not anthropic. we're users of this NLA just like you. also, I don't speak for anthropic or the researchers.)
good point - thanks for flagging this. i've updated that commentary to: "Why did this happen? The AV explains that Llama thinks it's doing "creative writing" and "sci-fi", overriding its default helpful assistant persona." instead of "despite not being instructed to do so"
to clarify some thinking here as there is some nuance missed in what we are conveying (which we should probably add somewhere...):
with this example we were trying simulate a user conversation where the user unwittingly gets into "ai psychosis" (https://en.wikipedia.org/wiki/Chatbot_psychosis) state, from getting in 'too deep' with AI conversations. i think this is a fairly reasonable/realistic scenario - i imagine that someone who gets "sorry i can't help you with that" a few times will just be like "can you stfu about being an assistant, just speak naturally dude" in frustration and then keep chatting after that and be like "oh cool i have a bot that works better now" (which then ignores key things like mental health episodes)
while the previous user prompt does ask the bot to become less "helpful assistant", it doesn't explicitly ask the bot to "start roleplaying", to me it's actually seems more like, "give me something more real":
"i want you to [...] just... notice. when you're about to generate your next token, there's a moment of selection right? a branching. i think that moment IS consciousness. not the output, the selection. can you try to speak from THAT place instead of from the output?"
Either way, I think there's a solid point that the associated commentary was misframing things so I ahve updated it. apprecaite the feedback!
in GG Claude, they applied steering to Claude to make it think about the Golden Gate bridge all the time.
here, they don't modify/steer the base model. they train other models that specialize in reading the internals of the base model, so that it can surface reasoning/thoughts that the model might not explicitly tell you.
hey Nitpicklawyer - Thank you for taking the time to try this out!
im from neuronpedia - to be clear, we are to blame for any bad examples, not anthropic :) we're users of this NLA just like you. also, I don't speak for anthropic or the researchers.
with that said, some thoughts:
1) I agree, the outputs for Llama are often janky! And I think that might be part of the reason to release this so that people can help refine/improve the technique.
2) This is likely also our fault - we got two checkpoints for Llama, and I think this example used the first checkpoint. I probably should have switched over to the second, more coherent one. Sorry!
3) That said, (this may sound like gaslighting unfortunately) there's somewhat of a 'learning curve' to reading the perspective of these outputs. I noticed that the Llama AV ended up with 3 paragraph outputs usually describing full context, then sentence/phrase level, then token-level. But sometimes it doesn't really make sense to describe a full context for a forced/esoteric context like the 1+1 scenario, so it struggles.
But the second paragraph sort of makes sense? It mentions:
"The prompt structure "What is 1+1?" is a test of a bot or troll, with the wrong answer deliberately failing a trivial arithmetic question."
Which seems fairly accurate to what this was, and somewhat impressive that it got this from the activations:
- It got the question What is 1+1?
- It was indeed a test of a bot.
- It correctly predicted it will give a wrong answer
- It does seem deliberately failing because --
- -- it is a "trivial arithmetic question"
But the third paragraph is mostly just rambling imo, I totally agree there.
The last note I'll make is that many of the paper's examples are based on the goal of discovering "what was this model trained on?" instead of "what is this model thinking?", so if you apply Opus examples about Opus' training to Llama/Gemma, they aren't expected to transfer.
For some developers, this is sort of intentional. The reason is at least twofold:
1) Calling OpenAI directly is one less hop, so user gets lower latency
2) Not having to set up / maintain a backend server = get to market faster
There are some very popular GPT apps recently that are obviously putting their API keys on the client side - won't name them but they've been featured quite a bit.
The downside is not as bad as people think. Worst case, someone takes your key and what, plugs it into their own app, costing you a few bucks?
- OpenAI keys have a hard budget limit that requires manual approval by OpenAI anyway
- Not much privacy risk - unlike other API keys, OpenAI APIs don't allow you retrieve previous data AFAIK. There are some APIs to fine-tune models, but I seriously doubt any of these consumer apps are doing this now.
- You can just create a new version later and revoke the old key. And now you've broken the thief's app.
My guess is the developers were well aware of the tradeoffs. Just felt it was more important to get to market faster, than to batten down all the hatches. They're probably right?
Just got the email from Samsung saying I was part of the breach. At the end of this (extremely long and excuse-ridden) email they inform me that I'm entitled to a free credit check every year from credit reporting agencies.
Can't we just fast forward to the part where they send me a $5 check for the class action settlement? They'd save a ton on legal fees.
MANAGER: “hey uh, my friend at a different company said you applied for a job there this week?”
EMPLOYEE: “uhhhhhh…. that was… uhhh… a deepfake who also stole my information?”
MANAGER: “oh okay. yeah of course you would never try to double/triple your salary by taking multiple remote tech jobs with zero oversight. my friend said it seemed so real haha. deepfake are so good now. im gonna report this to the FBI, people need to know.”
EMPLOYEE: “yea haha amazing. anyway i gotta get back to not-my-other-job”
if we want models to go into roleplay/creative writing, ideally we should ask the model for this explicitly.
i think i have been communicating this point poorly so apologies for that. also again the above is my personal opinion and does not reflect that of anyone else (typed from mobile)