This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
There’s debate over how much timing actually matters vs the rate of firing. Some people do believe in precise spike timing but I would say the general consensus is that spiking rate is a better measure of the current state of a system. There’s significant noise (as far as we can tell) in a neuron’s timing and it’s best modeled with a poisson process so we tend to think of it as rate coding which we can at least hand wave as viewing an RNN with a ReLU approximating.
Generally you can take a geometric view of this where certain features in a stimulus covary with neural activations in the same way they will with RNN “activations” which is at the real core of why people model things this way. The general idea being a dot product in an RNN can tell you something about what features are relevant for a task and we can look for hints of the same information being encoded in neural data. Certainly not everyone is in agreement on this though.
I feel like we’re all just operating on vibes at this point (including the government). These models are already powerful and could probably be used to do some bad things even without the next gen.
Anthropic has successfully made their vibe the bringers of the end of the world so please stop us. ChatGPT is in some weird middle ground where they’re no longer the evil company of a year ago and are trying to act like the mature one in the business. XAI is the vanity project of a jealous kid who wants to ignore anyone elses opinion but is falling behind. Gemini is viewed as the less cool cousin and no one pays attention to since google has generally avoided the end of the world talk (coincidentally I think gemini has the best non-coding model but no one seems to talk about that)
I think each company has been treated in the wider discourse based on their vibe and no one is operating on facts here since it’s all pretty obfuscated intentionally. I could see some things going very bad as a result of these models but I also don’t believe anyone who’s just pattern matching to their favorite scifi book here and we’re all playing into it. May as well go outside for a bit and enjoy the sun.
It’s not suppressed though, it’s just that the current evidence points in one way and much less in the other. There are plenty of review papers you can read to see how people weigh the evidence which you can of course disagree with:
https://doi.org/10.1016/0010-0277(95)00678-r
I think it’s less that it’s impossible but more that we don’t have any clue as to what causes differences among groups and many people use such measured differences as evidence for pretty deplorable ideas. There’s much more evidence for social determiners than anything biological. Everyone outside of Africa shares a single ancestor 20,000 years ago. There’s far more genetic diversity within Africa than the rest of the world. That alone is often enough to disprove many theories regarding racial differences since our intuitive understanding of “genetic difference” is so flawed.
Past research has a eugenicist bias because early statisticians were eugenicists seeking evidence for the ideas. I would argue that’s why the social determiners research is valuable to help offset that.
I see. I wonder how this works out in terms of risk/reward. I suppose if you take extinction as -infinite cost than it would be the only issue worth thinking about. Where I think this line of thinking gets challenging is when you need to take in terms of a counter factual. A lot of these were already risks prior to AI (bioweapons, nukes, etc) so what’s the marginal increase in probability as a result of AI I guess is the question which matters. I could get more around this way of framing it than saying that AI itself is the problem. It’s just the being more capable as a species increases risks. I think a lot of these pushback comes from the fact that it’s often the CEO who stands to gain huge by saying his tool is going to end the world so we need public buyin to supporting it. If instead it was just framed as “general technological advancement” is dangerous but potentially worthwhile I think more would be on board.
I wonder if our difference in view could be an instance of the jagged nature of AI’s intelligence. I do computational research in a basic science so write code or build models basically all day that is (occasionally) novel. I would say that I’ve noticed exponential improvements in parts of my job but certainly not all. For example, if I’m trying to visualize a concept from a paper I now go straight to Codex, give it the paper, and describe a webapp which allows me to play with the model in a way that wasn’t possible one year ago (this is great for teaching btw). If I have a script that I want to generalize, add in better metrics, or setup for running on a cluster I use codex and it does great.
Where it fails me though is exactly when I’m doing something novel like developing a new model or trying to develop some new method to process data. I’ve tried many times to one shot these ideas with detailed descriptions of what I want, how I’d like to generate abstractions, etc and it almost always ends up changing what I want to what I can only describe as something which better matches its training data. It often quietly changes key details that means that I have to delete the whole thing and start over. Just today this happened. On this level of task I’ve found that my workflow and pace of iteration hasn’t really changed at all in the last year. I still have to go and explain in detail on a function by function level what I want in much the same way I did a year ago. While that’s obviously a harder task, it seems to me like the task this whole long term exponential argument hinges on. I obviously could be wrong and maybe LLM with eval loop will do all of this for us but it seems still quite bad at anything without a clear definition of “good”.
I’m personally much more concerned about autonomous weapons, surveillance, and people plugging these things into places they don’t belong to avoid responsibility than I am the general possibility of these models being smarter than me in every way but obviously I could be wrong on this and am just using it incorrectly, hence the question.
That’s interesting. I commented something about this elsewhere but to me part of the exponential argument that loses me though is that it can often seem like a way to distract from issues that already exist which we should be working to fix. Things like autonomous weapons or mass surveillance are already here and rather terrifying and I would hope that we would dedicate our time to fixing those rather than having industry leaders focus so much on hypotheticals. While I guess the hypothetical scenario could be so bad that we must focus on it, I imagine a world which can’t come up with a way to spread wealth more equally or prevent mass proliferation of surveillance technology through profit seeking behavior will not be able to handle a digital super intelligence. So I keep coming back to the question: why is all I hear these industry leaders talking about is the threat of extinction? Maybe it’s just news coverage but I would love to see a leading lab release research on the health effects of subaudible sound in datacenters or other immediately present issues which would build good will towards these further out concerns.
I don’t really think it has to do with general denial in this case. I think he’s likely right in that pushback against datacenters is partially the release of angst over the threats which these models pose but in an indirect fashion since power is so unevenly distributed that local political organizing is the only real mechanism for people to act through. I think everyone feels nervous about these tools. People are scared of autonomous weapons, hackers using agents, and any of the other present issues that already exist. I think some of the pushback that comes in these threads is the fact that people like Dario are so focused on the long-term view that they suppress all immediate threats which already exist. You don’t need to argue about exponential growth or “AI being better than humans at everything” to want to better regulate this technology.
I would argue the best way to safeguard against long-term threats is to start by focusing on the issues that already exist. If you can offset the health risks of local datacenters or issues of unequal distribution of wealth by creating a more equal society right now then you’re already on the path to handling these long term issues. To me, this distant focus only distracts from the already present issues and conceals effective policy in this moment. We do need to safeguard against AI risk and it’s already here. Don’t even get me started on the havoc which recommendation systems have caused in society in the last 15 years which we still don’t call AI because it doesn’t speak.
Tl;Dr: These essays can feel disingenuous because:
1) AI risk is already present without exponential growth. The exponential growth argument often feels like a distraction from the fixes we could put in to fix the current issues that are already here.
2) The people stating this argument often have billions of dollars to gain if it comes true. While they may be altruistic, I also don’t see them doing all that much to fix the issues that people are already claiming exist and instead continue onwards on their path by justifying that they are the only ones responsible enough to handle it if the super intelligence does arrive. By continuing down that path, if that day ever does arrive they’ll have ensured the existence of a system which is unable to handle it.
I know this is likely just for IPO hype but when I read things like this I sometimes wonder if I must be missing something. I use agents everyday and find them really useful and they save me a lot of headache. At the same time I find that if I let it self-direct at a high level at all it generally makes bad choices that cause me headaches later so I can’t really give them autonomy. Enough people seem to believe this exponential line of thinking though that I keep having to wonder: am I the one missing something here? Is there some magic tool that I haven’t found yet that will cure cancer?
I’m more referencing the narrative that climate change is being used as a political point to enforce a global political state which stifles innovation (for example read the Project 2025 document around page 417 https://static.heritage.org/project2025/2025_MandateForLeade...). They do make some fair points in there but the primary change to the EPA that is being proposed is focusing on decentralizing the response to climate change to the states. In a vacuum where climate change is a matter of air quality this isn’t a crazy idea but the reality is that this is a global issue which is inconvenient if you want to prioritize growth/innovation over all else and the general idea behind climate conservationism is a powerful political force which the whole world is trying to shape right now for their benefit. There is right wing conservation which largely focuses on local, narrow issues (such as whale habitats being used as a reason to cancel offshore wind). There is also left wing conservation which pushes the more global climate change view and more large scale changes such as carbon taxes, increased stimulus for renewable energy, and often more regulation. Multiple things are true here though: 1. A lot of our regulation is pointless and serves to just increase cost and lock-in existing players as dominant forces in markets. This definitely stifles innovation so there is pushback against climate related changes for fear this will continue or worsen. 2. Climate change is a powerful force in modern politics which serves as a binding force for many disparate factions and is creating many grassroots movements. The impact of these on innovation isn’t clear (nor is it the point) but increased climate awareness is certainly increasing pushback against greenhouse gas emission intensive industries such as datacenters (in their current forms until renewable capacity can catch up) 3. Climate change is very real and is already increasing the cost of doing business and being less aware of it isn’t going to help. This will also stifle innovation as costs mount in the coming years.
I see this move as an attempt to stifle 2 by hiding 3. It seems like a bet that if we don’t focus on the global issues (like a collapsing current) then much of the political power behind climate change disappears since we are frankly quite lucky in the US to have a beautiful, and largely clean environment/air. Grassroots movements and politically powerful messages are likely an annoyance for those who want to keep doing business as usual since it benefits them to do so. Not trying to make a strong claim that our current response to climate change or what anyone in particular is doing is right or wrong but it seems clear that climate change is a powerful message that pushes for much grander scale changes than most people are comfortable with especially those with strong private interests in ignoring it.
This strikes me as one of the recent moves by our political/capital class where they think that if they just remove the information that’s inconvenient for them, people will stop caring and let them do what they want. You only need to listen to the bosses who so many of us work for to know that they think climate change is just an inconvenience in the way of progress. Only time will tell if this strategy will work or not.
I’m pretty skeptical that we’ll have actual convincing simulations of human brains any time soon. We’re still in the phase of trying to figure out how mice decide to turn left or right in a maze. Not to mention that the ethical/allowed ways of collecting human neural data are incredibly coarse especially the most common forms like fMRI or EEG (maybe some BCI will improve this but it’s still pretty rough technology). Most of our data as well is collected in incredibly controlled conditions (even “naturalistic” stimuli) so the data we operate on is still not particularly indicative of how people act in the real world. Maybe you mean simulations of a “mind” which behaves like people even if it’s not particularly accurate to the brain? What did you mean by this?
People is a broad term. Outside of major cities (where I live) libraries serve a very essential service for parents and their children and as a free communal space for the broader community. Our libraries are always full and a large part of the health of our area.
I recently moved over to codex after I couldn’t reup my membership and maintain access to clause code. I will say thus far I’ve found codex to work better and with less limits.
I think their point is that discounting the time estimates is more a constant shifting of the window of what we expect more than them being de-facto incorrect. They’re more off by degree (e.g. an XX% reduction vs complete extinction) than being worthless. As the example points out a large reduction can be very similar to an annihilation it’s just that we are only used to what we know so we constantly shift what is normal.