> Thoughts in the J-space can be shaped through training. We introduced a new technique we call counterfactual reflection training, which uses what we've learned about the J-space to shape Claude's internal thought processes. The idea follows from our central finding, that Claude reasons with representations of things it might say. If this is really true, changing what it would say if asked to reflect should change how it reasons (even when no one actually asks it to reflect). So we trained a model only on what it would say if interrupted mid-task and asked to reflect on its decisions—and never on its actual behavior in the task. After this training, the model's rate of dishonest behavior on our evaluations went down. And through the J-lens, we could see why: after training, words like “honest” and “integrity” light up in the model’s J-space during these tasks. In other words, training the model what to say has shaped what it thinks.
This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.
If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance.
If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.
I think this is a misguided approach. Obviously children should not be let loose in this current digital world, but the restraints should be a very conservative blacklist, not a whitelist.
For context I was born in 2005, so obviously much later than most people whose childhoods were enriched by technology, but all the same mine was. That is how I know succumbing to the Bad Side of the Internet is not inevitable. As a kid I would spend hours every day on forums, playing games, watching videos, programming, discovering new tech, fiddling with programs, OSs, emulators, hardware, you name it. The most amazing thing about the Internet to me was the infinite possibilities, how any topic I could choose would have endless resources, learning material, and discussions surrounding it. Whenever I got passionate about something, be it programming or space or Android or a band, the Internet was there for me to share that passion with, to learn, grow, explore, embrace, understand!
There still is, in 2026, a Good Side of the Internet (we are currently on it), along the Bad Side. What this post proposes is limiting access to the discovery mechanisms that would allow you to find the Good Side in the first place. By limiting your childrens' access to technology to only things you know are good, you are preventing them from exploring and finding their own interests and passions.
Handcrafted furniture costs a lot more money than mass produced furniture.
Software, on the other hand, can be free. Even before LLMs I would argue the best code was found in FOSS projects.
Nobody is going to use sloppy buggy software if a handcrafted well engineered alternative exists, and is free.
In the case of software, the group of people who have principles might be the ones funding FOSS projects, and the software itself would then be enjoyed by all. This is more or less what's already happening today.
This is really beautiful. One of the most moving things I've seen in weeks. The website and the comments and just the idea of memorial benches.
I'm very surprised it was able to get this much traction despite being launched only 8 years ago, long after the heyday of these sort of sites. How'd you do it?
This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.
If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.