Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.
To me, the clearest way to think about signed vs. unsigned integers is that different representatives of the integers modulo n are chosen.
For example, for 8-bit signed integers we choose the representatives -128, -127, ..., 127 of the residue classes -128 + 256Z, -127 + 256Z, ..., 127 + 256Z in the ring of integers modulo 256.
For 8-bit unsigned integers, we instead choose the representatives 0, 1, ..., 255.
Mathematically, I do not see how anything is "breaking" as the article claims.
Of course. Theoretically, the determinant answers the binary question "singular" or "nonsingular".
Numerically, such a binary answer is pretty useless. Here, we need a measure of how singular/nonsingular a matrix is relative to the numerical precision we are working with.
Edit: Sorry, I completely messed up my original answer here. A better version:
Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying ring. The only invertible elements in Z are -1 and 1.
So, the code is also incorrect in the integer setting. Here, we should not check for 0, but for -1 or 1.
They define a generic "is_singular" function and test it with a 2x2 matrix.
The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it away is that the determinant is influenced by a rescaling of the matrix:
det(s A) = s^n det(A) where A is a n x n matrix
As an example, would you say that
[[1e-10, 0], [0, 1e-10]]
is singular? It has condition number 1.
From page 6, when they describe their synthetic textbook dataset:
"Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]"
No. The determinant is not a suitable way to do that. A proper way to numerically measure singularity would be to compute the condition number of the matrix (the ratio of its largest to smallest singular value).
GPT-4's explanation of its optimization does not make sense to me. It writes
"Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that mov had happened.
AlphaDev's optimization relies on the fact that B and C are already in the correct order. This precondition is missing from the prompt given to GPT-4. It seems that GPT-4 is hallucinating something that only resembles the correct optimization at first glance.