It depends on how you see it. I agree with you, generally, but in the limit, if you memorised all possible instances of multiplication, then yes, you could certainly be said to know multiplication.
I've not just come up with that off the top of my head, either. In PAC-Learning (what we have in terms of theory, in machine learning) a "concept" (e.g. multiplication) is a set of instances and a learning system is said to learn a concept if it can correctly label each of a set of testing instances by membership to the target concept with arbitrary probability of error. Trivially, a learner that has memorised every instance of a target concept can be said to have learned the concept. All this is playing fast and loose with PAC-Learning terminology for the sake of simplification.
The problem of course is that some concepts have infinite sets of instances, and that is the case with arithmetic. On the other hand, it's maybe a little disingenuous to require a machine learning system to be able to represent infinite arithmetic since there is no physical computer that can do that, either.
Anyway that's how the debate goes on these things. I'm on the side that says that if you want to claim your system can do arithmetic, you have to demonstrate that it has something that we can all agree is a recognisable representation of the rules of arithmetic, as we understand them. For instance, the axioms of Peano arithmetic. Which though is a bit unfair for deep learning systems that can't "show their work" in this way.
Mwell, the article claims, and points to work that also claims, that large language models can actually be made to perform arithmetic well. They need fine-tuning, verification, chain of thought prompting and majority voting to be combined but the linked Google blog says that Minerva hit 78.5% accuracy (on the GSM8K benchmark).
For me the problem is that we can look at the output and say if it's right or wrong, but we know what language models do, internally: they predict the next token in a sequence. And we know that this is no way to do arithmetic, in the long run, even though it might well work over finite domains.
Which is to say, I'm just as skeptical as you are, and probably even more, but I think it's useful to separate the claim from what has actually been demonstrated. Google claims its Minerva model is "solving maths problems" but what it's really doing is predicting solutions to problems like the ones it's been fine-tuned on, and those problems are problems stated at least partly in natural language, not "naked" arithmetic operations. In the latter, language models are still crap because they can't use the context of the natural language problem statement to help them predict the solution.
Btw, "chain of thought prompting" if I remember correctly is a process by which an experimenter prompts the language model with a sequence of intermediary problems. So it's not so much the model's chain of thought, as the experimenter's chain of thought and the experimenter is asking the model to help him or her complete their chain of thought. I have a fuzzy recollection of that though.
> It's a language model; why would we expect it do math or try to somehow shoehorn math into the model?
Language models can do math, or anyway arithmetic. That's because language models are trained to predict the next token in a sequence and an arithmetic operation can be represented as a sequence of tokens.
The only problem is that language models are crap at arithmetic because they can only predict the next token in a sequence. That's enough to guess at the answer of an arithmetic problem some of the time but not enough to solve any arithmetic problem all of the time.
More generally, the answer to your question is in the same Figure 3.10 I've referenced above. OpenAI (and others) have claimed that their large language models can do arithmetic. So then people tested the claim and found it to be a bag of old cobblers.
Hence the article above. Nobody's trying to "shoehorn" anything anywhere. It's just something that language models can do, albeit badly.
I've not just come up with that off the top of my head, either. In PAC-Learning (what we have in terms of theory, in machine learning) a "concept" (e.g. multiplication) is a set of instances and a learning system is said to learn a concept if it can correctly label each of a set of testing instances by membership to the target concept with arbitrary probability of error. Trivially, a learner that has memorised every instance of a target concept can be said to have learned the concept. All this is playing fast and loose with PAC-Learning terminology for the sake of simplification.
The problem of course is that some concepts have infinite sets of instances, and that is the case with arithmetic. On the other hand, it's maybe a little disingenuous to require a machine learning system to be able to represent infinite arithmetic since there is no physical computer that can do that, either.
Anyway that's how the debate goes on these things. I'm on the side that says that if you want to claim your system can do arithmetic, you have to demonstrate that it has something that we can all agree is a recognisable representation of the rules of arithmetic, as we understand them. For instance, the axioms of Peano arithmetic. Which though is a bit unfair for deep learning systems that can't "show their work" in this way.