You have a point. I would further add that private ownership of these things requires capital concentration that cannot be healthy for society.
On the other hand, are we replacing public with private infrastructure here or is the private sector filling gaps where we didn't have any public infrastructure before in the first place?
The end of the UN seems plausible if we continue to move from a rule-based world order to a power-based one. For the same reason, there is little chance we will see the end of the EU. No European country is sufficiently powerful to really matter on the world stage, a more united Europe has the potential to be a significant world power though.
The scaling laws are also power laws, meaning that most of the big gains happen early in the curve, and improvements become more expensive the further you go along.
LeCun has already proved himself and made his mark and is now in a lucky position where he can focus on very long term goals that won't pay off for a long time (or ever). I feel like that is the best path someone like him could take.
I would say GPT itself is less an event and more the culmination of decades of research and development in algorithms, hardware, and software. Of course, to some degree, this is true for any novel development. In this case, the convergence of development in GPUs, software to utilize them well while being able to work in very high levels of abstractions, and algorithms that can scale is something I'm not sure we will see again so quickly. All this preexisting research is kind of a resource that will be completely exploited at some point. And then the only thing that can drive you forward are truly novel ideas. Reasoning models were a fairly obvious next step too as the concepts of System 1 and 2 have been around for a while.
You are completely right that the compute and funding right now are unprecedented. I don't feel confident making any predictions.
Basically what we have done the last few years is notice neural scaling laws and drive them to their logical conclusion. Those laws are power laws, which are not quite as bad as logarithmic laws, but you would still expect most of the big gains early on and then see diminishing returns.
Barring a kind of grey swan event of groundbreaking algorithmic innovation, I don't see how we get out of this. I suppose it could be that some of those diminishing returns are still big enough to bridge the gap to create an AI that can meaningfully recursively improve itself, but I personally don't see it.
At the moment, I would say everything is progressing exactly as expected and will continue to do so until it doesn't. If or when that happens is not predictable.
If you can't manage it every day, a less frequent habit will still have benefits.
I've been doing strength training once a week since the pandemic startet, and it has been great. Obviously I could have made much more progress with more frequent training, but I still look much better naked and more importantly, I feel much better. So many random aches and pains have just disappeared.
In my experience with a fairly recent and expensive set of bluetooth headphones (Sennheiser Momentum 3), it works well most of the time, but in 5% of cases it does not. If you use your headphones a lot, those 5% get annoying very quickly.
Some issues I've noticed:
- My tv (flagship LG OLED) always has trouble connecting to the headphones. I usually have to turn them off and then back on again at least once. Macbook and iPhone rarely have trouble connecting. Connecting on Windows has always worked so far.
- When I'm connected to both my Windows laptop and my phone, in a skype conference on the laptop and get a call on my phone, the skype conference is automatically terminated. FOR ALL PARTICIPANTS. That's insane.
- To make the previous point even worse: except for the ringtone, the audio from incoming calls never even goes to the bluetooth headphones. The audio from outgoing calls does, however. This is all on an iPhone SE.
- Sometimes when I listen to music, or watch a video on a Windows machine, my bluetooth headphones will just pause the playback every couple of seconds. This only happens on Windows.
- If your headphones are connected to multiple devices, you have absolutely zero control over what is happening. You might be listening to music on your phone, but some notification from your computer will interrupt your playback even though the computer is set to silent.
I could go on and on and I've only been using bluetooth headphones for a month. There is a LOT of polish missing here. That being said, this could be a great technology and it's not too far off at the moment. There certainly are some advantages over wired headphones as well.
On the other hand, are we replacing public with private infrastructure here or is the private sector filling gaps where we didn't have any public infrastructure before in the first place?