It's mentioned briefly in the paper(1), but I'm more interested in the interpretability implications of this approach. In some respects, this marries the interpretability/editability of a small decision tree with the expressive power of a large neural network. Usually you see those two on extreme opposite ends of a tradeoff spectrum - but this approach, if it scales, might shift the pareto frontier.
(1): As a byproduct, the learned regions can also be used as a partition of the input space for interpretability, surgical model editing, catastrophic forgetting mitigation, reduction of replay data budget, etc..
This is a fascinating article, but I'm also very impressed by how well-communicated and transparent the research is. The writing is impressively clear, simple, and precise. The interactive feature explorer (https://transformer-circuits.pub/2023/monosemantic-features/...) is awesome, and lets me explore the claims of the article myself. The background information filled in a lot of gaps for me where I haven't been following the research closely.
I would love to see if this analysis can scale to a multi-layer transformer network, there are so many interesting questions to answer!
- For single-layer transformers, the inputs are text and the outputs distributions over text, so we can directly inspect and interpret the strings that the features are sensitive to, and the predictions the features make. With two layers, that isn't the case any more for the intermediate bit between the two layers. I guess the question here is... can we transfer this approach to a multi-layer context at all, or does it fundamentally not make sense?
- Are second-layer features fundamentally different in character from first-level ones? How often do we see the same feature in multiple layers, iteratively refined, vs categorically different features?
- How does the existence of the second layer change the kinds of features learned by the first layer?
- How do features evolve throughout the training process? I often imagine features getting 'compressed downwards' as training progresses, requiring N levels to fully resolve at epoch t, and perhaps n - 1 levels at epoch t + k. Can this approach give us a comprehensive picture of how and when that happens? Could we actually track a single feature as it shifts from the second layer to the first? The bottom-up nature of this approach - vs top-down looking for some specific expected feature - is really compelling here, as it could let us get a sense for how features behave in this regard on a population level.
- Could we identify causal relationships between first-layer features and second-layer features with some extension of this approach?
- Combining the above two points, I could imagine a 'meeting in the middle' of low-level features and high-level features: if both take some span of layers to be computed, and they're iteratively refined across that layer span, and at some point the spans become disjoint; this effect could explain some of the 'phase transition' like behavior in the training process. It would be amazing to be able to directly observe this occurring!
Intervening in the direction of a feature seems very powerful for safety, as mentioned. However, because of the interference effects from the overcomplete basis, one might expect other unrelated features to be affected/degraded in subtle ways if we naively intervened on a feature. I wonder if this feature-identification process could be incorporated into training the model somehow - periodically identifying safety-relevant or otherwise important features, and encouraging just those particular features to be disentangled/orthogonal/sparse/aligned with the neuron basis? I'm not sure if this even makes mathematical sense, my linear algebra is quite iffy, but it seems potentially quite interesting to learn a representation space where the basis is somehow a mix of complete (for the 'privileged' features) and overcomplete (for everything else).
Sorry for the wall of text, this is just very exciting stuff and I had to get my thoughts out through my fingers somehow!
Oh yeah, I empathize with that frustration! I've come to the conclusion that LLM applications require a fundamentally different approach to reliability than we're used to. Traditional programming is about composing abstractions to build more complex abstractions; because the low-level abstractions are so close to perfect, we can build these immense towers of abstractions and still have effective guarantees on their behavior. With LLMs, perfect abstractions are impossible, and composing them naively will exponentially magnify their imperfections.
Instead, I think composing LLMs needs to be done in a way that degrades gracefully, with resilience to failure being a fundamental consideration. Biology has similar properties; complex biological systems (ecosystems, cells, etc) have feedback loops, redundancy, and most of all diversity. If we take a similar approach to building LLM apps, we'll end up with things like:
- multiple different prompts used in parallel, with results joined e.g. with voting. A change in how one prompt behaves can thus only have a bounded effect on the system as a whole.
- some way for an LLM to productively express 'this thing you're asking me to do is nonsense', with monitoring and continuous evaluation hooked up to that signal, and maybe runtime retry behavior as well. This can help with when you get into situations where prompt A gets an "I'm afraid I can't do that" response, and then you give that to prompt B as if it is a valid thing, and that cascades through the rest of the application.
llmtaskgraph as a library is designed to make building, operating and maintaining systems with these sorts of features easier - without good observability, it's impossible to know if some feedback loop is doing its job, or which prompts in a pool are behaving well vs poorly, much less what effect they are having on the rest of the system.
Sorry for the wall of text, I got a bit nerd-sniped. :)
This is a really natural extension of CoT. I was experimenting for a month or two with a similar concept in a hobby project this past spring: https://github.com/knexer/llmtaskgraph . I'm really excited to see more people exploring in this direction!
I was focusing more on an engineering perspective; modeling a complex LLM-and-code process as a dependency graph makes it easy to:
- add tracing to continuously measure and monitor even post-deployment
- perform reproducible experiments, a la time-rewinding debugging
- speed up iteration on prompts by caching the parts of the program you aren't working on right now
My test case was using GPT4 to implement the operators in a genetic algorithm, which tbh is a fascinating concept of its own. I drifted away after a while (curse that ADHD) but had a great time with the project in the meantime.
I like the first-order vs second-order distinction here - this is a clean way to describe something that I've often found hard to communicate to others, at least for those familiar with functional programming. Everyone's familiar with first-order use of a language model at this point (it's just plain chatgpt) but higher-order use seems much more difficult for most to even conceptualize, much less grasp the implications of.
The huge challenge with higher-order use of LLMs is that higher-order constructs are inherently more chaotic - the inconsistency and unreliability of an LLM compound exponentially when it's used recursively. Just look at how hard it is to keep AutoGPT from going off the rails. Any higher-order application of LLMs needs to contend with this, and that requires building in redundancy, feedback loops, quality checking, and other things that programmers just aren't used to needing. More powerful models and better alignment techniques will help, but at the end of the day it's a fundamentally different engineering paradigm.
We've been spoiled by the extreme consistency and reliability of traditional programming constructs; I suspect higher-order LLM use might be easier to think about in terms of human organizations, or distributed systems, or perhaps even biology, where we don't have this guarantee of a ~100% consistent atom that can be composed.
Half-baked aside: in some ways this seems like a generalization of Conway's law (organizations create software objects that mirror their own structure), where now we have some third player that's a middle ground between humans and software. It's unclear how this third player will fit in - one could envision many different structures, and it's unclear which are feasible and which would be effective.
(1): As a byproduct, the learned regions can also be used as a partition of the input space for interpretability, surgical model editing, catastrophic forgetting mitigation, reduction of replay data budget, etc..