It's actually not linear, its a power law. That means we need exponentially more compute, data, and model parameters to see linear improvements in performance.
Not to pour too much cold water on this, but the claim of 100% accuracy has a huge caveat. In the paper (Page 4) they state:
Interaction. The original question may not be a prompt that synthesizes a program whose execution results in the correct answer. In addition, the answer may require multiple steps with clear plots or other modalities. We therefore may interactively prompt Codex until reaching the correct answer or visualizations, making the minimum necessary changes from the original question
Which to me basically sounds like they had a human in the loop (that knows how to solve these math problems) that kept changing the question until it gave the correct answer. They do measure the distance (using a sentence embedding model) of the original question to the one that yielded the correct answer, but that feels a bit contrived to me.
Nevertheless, its still really cool that the correct answer is indeed inside the model.
It looks like this is just Pilocarpine, which has been used for decades to treat glaucoma, available at pharmacies everywhere, and is commonly used (perhaps off label) to shrink pupils. I wonder what if anything they changed compared to the generic version?
I've been using Pilocarpine off label to shrink my pupils at night after ICL surgery (an alternative to lasik) to solve debilitating halos caused by my pupils growing larger than the implanted lens.
In my experience, it does increase close range vision (at some minor expense to long range vision). That said, it also gives a mild headache, and blurs your vision substantially for the first 5-15 minutes after use. I don't really see the appeal of using it daily unless you really have to.
While I sort of agree that machine learning will end up as an experimental science, it's way, way too early to say whether the theory relating deep learning to kernel methods (e.g. Neural Tangent Kernels) will be useful or not.
As an example, just last week a (huge) paper [1] was put on arXiv that used these theoretical methods to analyze a bunch of common architecture building blocks (skip connections, normalization, etc), and then applied their theoretical findings to figure out how to train Resnet like models in similar training time without these seemingly "required" building blocks.
Deep Learning is still in its infancy in many ways, and this type of research takes time, slowly building on successive results.