> what kind of neural-net architecture and training would allow a model to handle numbers lengths it hasn't been trained on
A recurrent neural network implementing binary addition with carry could do this, and one can derive the correct weights with pen and paper without too much effort.
Whether gradient descent will find them too is another matter entirely
I had similar thoughts, and my conclusion is that competition is an inherently unstable state of affairs: at some point somebody wins, and they will try very hard to prevent any further competitors from arising.
Indeed, competition is undesirable for all participants involved: everybody wants to win and exploit the rest for their own gain. Note that this is the only way to make competition work and result in its temporary benefits (if nobody wants to win, nobody will compete).
So there must be a system to keep the competition going and preventing the rise of a definitive and exploitative winner, and the existence of this system has to be accepted by the competitors. But why would serious competitors accept a system that prevents them from winning?
It is both common and uncontroversial to put restrictions on using certain tools in certain situations for safety reasons, especially in public and crowded places: you can't bring a hammer to a concert.
As the provider of a public place, X ought to take certain measures to ensure public safety on its premises. Of course, deciding what is and in not tolerable is the crux of the issue, and is far from trivial.
No, this would crash with numpy arrays, pandas series and such, with a ValueError: The truth value of an array with more than one element is ambiguous.
> how important is computing the exact gradient using calculus
Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance https://arxiv.org/abs/2202.02831 (and many other works relating to SGD noise)
> vs just knowing the general direction to step
I can't find relevant papers now, but I seem to recall that the Hessian eigenvalues of the loss function decay rather quickly, which means that taking a step in most directions will not change the loss very much. That is to say, you have to know which direction to go quite precisely for an SGD-like method to work. People have been trying to visualize the loss and trajectory taken during optimization https://arxiv.org/pdf/1712.09913 , https://losslandscape.com/