Yes - we'd never normally turn features up this much as it breaks the model quite badly, but we put this in the post to show what that looked like in practice.
Thank you! I think some of the features we have like conditional steering make SAEs a lot more convenient to use. It also makes using models a lot more like conventional programming. For example, when the model is 'thinking' x, or the text is about y, then invoke steering. We have an example of this for jailbreak detection: https://x.com/GoodfireAI/status/1871241905712828711
Being able to read features out and train classifiers on them seems pretty useful - for instance we can read out features like 'the user is unhappy with the conversation', which you could then use for A/B testing your model rollouts (kind of like Google Analytics for your LLM). The big improvements here are (a) cost - the marginal cost of an SAE is low compared to frontier model annotations, (b) a consistent ontology across conversations, and (c) not having to specify that ontology in advance, but rather discover it from data.
These are just my guesses though - a large part of why we're excited about putting this out is that we don't have all the answers for how it can be most useful, but we're excited to support people finding out.
No images in training - 3.3 70B is a text-only model so it wouldn't have made sense. We're exploring other modalities currently though.
SAE is a basic ReLU one. This might seem a little backwards, but I've been concerned by some of the high-frequency features in TopK and JumpReLU SAEs and the recent SAE (https://arxiv.org/abs/2407.14435, Figure 14), and the recent SAEBench results (https://www.neuronpedia.org/sae-bench/info) show quite a lot of feature absorption in more recent variants (though this could be confounded by a number of things). This isn't to say they're definitely bad - I think it's quite likely that TopK/JumpReLU are an improvement, but rather that we need to evaluate them in more detail before pushing them live. Overall I'm very optimistic about the potential for improvements in SAE variants, which we talk a bit about at the bottom of the post. We're going to be pushing SAE quality a ton now we have a stable platform to deploy them to.
The paper is measuring I/O behaviour, rather than the complexity of the mechanisms generating that behaviour. Transistors might have quite complex physics, but are designed to have relatively simple I/O behaviour.
I'm not sure that's all there is to it - at least some scientists are in it for the explanation: they want to know more about something. In this case the prediction is just a useful check that their proposed explanation is compatible with reality.
Of course, this isn't typically why science gets funded - we want the engineering applications that are enabled by our ability to calculate - but a version of science that's all prediction, no explanation seems very unappealing (not to mention sterile for further investigation).
Modern Classical Physics is a great book - I've recently started working through it (just about to move on to chapter 2). I'd be interested in chatting about it with others/cross-checking solutions. Anyone who's interested, drop me an email (address in my profile).
If you're thinking of getting it but want to check it out, there's a 2012 draft version that the authors have previously taught from here: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/. It's not the same as the book, of course, but from a skim it seems quite similar.
As far as I know, a good differential-geometric understanding of nonequilibrium thermodynamics still hasn't been achieved.
The central issue is understanding how changes in control parameters (for instance concentrations of catalysts in a chemical system, or local fields in a spin system) affect the evolution of the probability distribution over states. Some work has been done in close to steady state (for instance [1,2,3]) but it's far from resolved.
This has some nice applications - designing efficient protocols for microscale devices, for instance.
As I understand it her point it that what we take to be fundamental particles have certain properties - charge, spin, etc - and no others. If they had other properties, this would lead to measurably different outcomes in collision experiments for quantum-mechanical reasons (not because of any 'free will' on the part of the particles, as I believe some other commenters have interpreted it).
If these few numbers really are all the information that an electron (for instance) contains, then where is the informational content of consciousness located, assuming that panpsychism claims that electrons possess a certain small amount of consciousness? This is how I interpreted her sentence on consciousness implying the ability to change; not as meaning the ability to decide, but as meaning the ability to carry extra information by being in different states. I think this seems like a reasonable objection and I'm interested in how panpsychists might respond.
I think the standard reference is probably Spivak's 'Calculus on Manifolds' but this never really did it for me.
If you have a background in physics then some combination of Nakahara's 'Geometry, Topology and Physics' and Baez and Muniain's 'Gauge Fields, Knots and Gravity' might be good (I haven't included relativity textbooks as I assume it you have a background in GR then you have enough differential geometry).
An unusual recommendation that I think is really nice is 'Stochastic Models, Information Theory and Lie Groups' by Chirikjian. It covers a few other topics mentioned in this thread and is really nice. It's _extremely_ concrete and spells out a lot of calculations in great detail. Plus, the connection to engineering applications is much more obvious.
In this case that's not true though - the article is about someone who used critical thinking/reasoning skills to make a huge amount of money, and credits this in large part to his training in philosophy.
It's possible, of course, that Miller is mistaken about the origin of his critical thinking ability (or just got lucky, and there's no ability to credit), however on balance this seems like a strong piece of anecdata in favour of the hypothesis.
I'm interested what would constitute empirical evidence for/against the claim "philosophy helps improve critical thinking". Are there accepted standardised measures of this?
I'd like to flag up this paper on HMC and minibatch inference: https://arxiv.org/abs/1502.01510v1. It seems that SGHMC might not always behave as well as you might like.
I should say this isn't to pour cold water on the idea of Bayesian deep learning - I think it's an extremely sensible idea and undoubtedly a good direction for research, but just wanted to caution that this particular MCMC method might have subtle flaws.
I agree - it's a surprising and cool paper. There has been some work done on fooling network ensembles by constructing constructing a Bayesian posterior over weights using dropout [0]. This is an ensemble of weights for the same network, not over different architectures, however.
The basic idea here is that most of the time, each member of the ensemble will misclassify the adversarial example in a different way. This means that the posterior predictive distribution for adversarial examples ends up much broader, and you can detect them this way.
Surprisingly, even this can be beaten in the white-box case [1], although it's by far the hardest to beat of the current adversarial defences, and needs much more distortion. It's beaten exactly as the GP says, by differentiating through the averaging. AFAIK no-one's tried architecture ensembling, but I expect it would be vulnerable to the same technique.
You are absolutely right that a lot of work to date (especially in applications and metrics-driven research) has focused on 'functions for vision problems'. But this is far from the only idea - that's the point I'm trying to make.
For instance what about learning value functions for reinforcement learning (e.g. AlphaGo)? Or natural language processing? These are definitely not vision problems, or if you believe that they are then 'functions for vision problems' is actually a pretty huge class!
The universal approximation theorem backs up my claim [0] - we can approximate arbitrary functions with neural networks. I think this theorem is overemphasised in practice: we don't generally want to approximate arbitrary functions, we _want_ to encode specific prior information into the function we approximate, as you rightly say. But that doesn't mean that we have to do so, or that we only have one idea about what functions to encode, or even how to encode them.
As commenters elsewhere have pointed out, this really only caps salary at around £60k - this is actually pretty low for an experienced ML engineer.
I've heard Prof Pantic advocate for salary caps for PhD students leaving academia before and just can't understand it. It could mean that _all_ jobs must be capped - this seems like an overreaction to market dynamics in one corner of computer science. If not, is she advocating for salary caps for PhD students exclusively? This would be almost trivially easy to game and also downright exploitative.
In addition, we're told almost constantly how few academic jobs there are, and how we'll all have to leave academia. As soon as this starts to actually work well, suddenly it becomes a problem.
I've tried to construct a strong/charitable version of this argument before and failed - does anyone have one?
I suppose it's limited by the size of 'the set of things you can do with functions'. This seems like a large enough set that I wouldn't be worried about hitting its limits too soon...
There's a reason the axes are left unlabeled in this graph. If you look at a paper that gives values over time for these quantities you'll find that this 'effect' mounts up to about 3 IQ points over the range discussed in the article in a population with an SD of 12 [0].
As a basis for discrimination, this is nonsense: on an individual basis you learn next to nothing about an individual's IQ if you know their age. Of course, there may be better studies out there - I'm not an expert and so don't know of them.
There's also an interesting set of applications in statistical physics: many systems can be modelled by Markov chains whose rate parameters can change through time. Changing these parameters non-quasistatically (i.e. so that the system does not relax to the appropriate steady state) gives rise to distances on parameter space that are related to the amount of energy dissipated along the path through parameter space [0,1].
What's nice about this is that the derivation of a suitable metric allows us to compute trajectories that minimise quantities we care about (e.g. minimise energy dissipated), so this has clear potential to be useful. Some cool examples are in spin systems [2] and a harmonic trap [3].
For any differential geometers reading this: it seems to me that a good geometric way to think about this is as a fibre bundle, with the parameter space being the base space and the simplex being a vector bundle over it (see [4] on the simplex being a vector space).
It runs through a lot of important topics, particularly in inference, without being either as turgid as most stats texts for people without a maths background or as dry as more 'pure' books (no measure theory required).
For biology I'd recommend 'Physical Biology of the Cell' (http://microsite.garlandscience.com/pboc2/) if you like to think quantitatively. About evolution specifically I find Schrodinger's 'What is Life?' thought-provoking if you already know the basics.