this guy seems so full of himself. Everything I read of his triggers my bullshit alarm. Stuff like claiming feasible solutions to problems that have been mathematically proven don't have any
I think of it like this. Imagine a network with two inputs and one output. What's happening during training is to orient a set of 2d planes in 3d space. Then for each x and y coordinate you can iterate through those planes and figure out where the normal at (x,y) hits them on the z axis, take the highest of your result and that's your network output.
Sometimes the solution needs one of the planes to make a big change to its orientation. and in the process of doing so the fitness will go down. Now if some other plane happened, by luck, to be in a better position to get to that state without lowering the fitness, then it will generally dominate over the other one, and the first one would start to "evolve" to be suppressed by the other one. The first one then becomes (usually, but not always) pretty much dead. Since the network has a fixed number of weights, it can't decide to just "add another plane". It has to make do with a fixed number of them. If too much of them become useless your training can't really recover, you're stuck at that minimum. The overparametrization allows for a bigger chance that some other plane's orientation will happen to be in a position to shadow counterproductive ones.
I think a language having a central repository of libraries that anyone can publish stuff in is a stupid idea in the first place. It's not a question of if, it's a question of when it goes to shit.
Why anyone would even consider using an online password manager is beyond me. Keepass works perfectly offline, and it's easy to sync. Just copy one file. I set up syncthing on my pc and phone and I basically just forget about it. For other places I have a particular subdomain (guid) where I serve the current copy of the file, gated by a password. The database itself is protected by both a password and a yubikey.
you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"
If they're to be trated as disposable, then they should be treated as disposable