If your "proof" can't port to Humans then it's not proof
Learn to take a hint. I'm not going to argue this on human terms because you're playing a dumb um-akshually game.
Computer reasoning systems can solve vastly more complex problems perfectly. Expert mathematicians can solve vastly more complex problems with only minimally increased errors. The ability of LLMs to solve reasoning problems completely disintegrates when the problems get more complex.
Trying to argue that LLMs are alike humans because of you can put these three into the buckets of "No mistakes" and "Some mistakes" is ridiculous.
Nobody is calling LLMs perfect reasoning machines.
Yes.
You said humans make mistakes, my point here is, humans make mistakes precisely because they stop doing reasoning and start doing blind pattern matching estimation of the answer.
The idea that you must make no mistake reasoning before you can be considered to be reasoning has no ground.
Reading comprehension.
I did not say no mistakes. I said that the failure pattern follows that of estimated guesses; Rapidly increasing errors as the size of the problem increases.
Whereas with computer reasoning, the rate of errors does not increase at all. And with (expert) humans the rate only goes up a little.
Did you even bother looking at the link?
You are missing the point.
I am not referring to literally English or any other language. I'm referring to the structure of language problems, which is vastly simpler than any moderately complex math or programming problem.
To more explicitly spell out the reason for my unimpressed-ness: They trained a pattern-repeating-machine and found that it will repeat some of their patterns, some of which were patterns trained on.
This does not demonstrate the ability to reason abstractly about new models, so I do not care.
It's an impressive result, but shouldn't be seen as "correction". Framing it as a (drastic) reduction in mistakes is more useful here.
If the model is productionized (read: dumbed down so it isn't as expensive to run), the reasoning abilities drastically decline again.
And these reasoning abilities are still around a language model, rather than around abstract models.
This is a very effective party trick for general math, whose language quite directly maps onto these abstract concepts, but there are some holes. Information about e.g. which values may be zero isn't encoded in the language, and so this approach is liable to blundering around division-by-zero issues.
If you want a particular example to toy around with, LLMs are not fond of quaternions and their conversion to other representations.
Humans provide increasingly wrong answers as questions get more complex too.
Human this, Human that. LLMs aren't humans. "My model is crap but the human brain isn't very good at this either" is irrelevant when we have machines that are not only very good at these tasks but almost perfect at them.
Humans make such mistakes precisely because they are not perfect reasoning machines. To compare LLMs to humans is not only disingenuous, but proves my point.
(And no, I will not humour you with an argument about how the amount of wrong answers is drastically lower from human mathematicians)
Jumping from that to "incapable of abstract reasoning" is silly.
They are language models. It is explicitly what they are designed to do.
If these LLMs are not, as I claim, reasoning on language rather than the abstract model of the query, then how come they fail miserably in exactly the ways you would expect where that the case?
LLMs generalize to non linguistic patterns.
Yes, congratulations, if you turn a problem into a linguistic one LLMs can deal with them. This does not in any way go against what I said about the capabilities of LLMs.
The same levels of actual abstract reasoning can be achieved on a graphing calculator running off literal potatoes.
This is where the terminology becomes a bit annoying, but there is a key difference in the kinds of reasoning at work here.
When you ask LLMs to provide a reasoning, the actual reasoning performed is linguistic; The LLM has (is) a model about language and performs some (limited) reasoning on that model to get an output.
But that is explicitly different from reasoning about the abstract question at hand, thus the answer is mostly a guess.
The key difference to observe is that "semantic reasoners" like computer algebra or prolog, always maintain correctness within the axioms provided. They may slow down significantly as questions get more complex, but they do not start providing wrong answers. Computers are flawless mathematicians, provided they are programmed correctly.
LLMs do provide increasingly more-wrong answers as the question gets more complex. Thus we can observe that LLMs do not abstractly reason about the question and it's model.
You're missing the point, there is a difference; The answers are often wrong, and more-wrong the more complex the question gets.
They're only able to answer simple (relative-to-the-model's-size) straightforward reasoning questions. Which is a nice party trick, but not broadly useful.
They can however tell you how to convert that problem into steps that can be run in an algebra system.
Usually they can't do that very well either. Converting a problem from one description to another is algebraic reasoning, subject to the issues already mentioned.
What they can do is summarize general instructions and documentation, provided adequate training data was available.
They're neither trained on such problems, nor is that a goal for LLMs
Yes. But LLMs keep being pushed for tasks that heavily involve abstract reasoning, which is dangerous as they're unsuited for it. (E.g. Any code generation that isn't mere empty boilerplate.)
Given a set of instructions, an instruction fine-tuned/aligned LLM is able (conditional on size and training quality) to reason through a set of steps to produce a desired output.
This is plainly wrong. The model's growing size makes it better at guessing the outcome of a reasoning task, but little to no actual reasoning is performed.
It's trivial to prove this as well, as LLMs will still fail miserably at (larger) math problems that even basic computer algebra systems will handle with ease.
The issue with them is that it's not simply "looking at oneself".
If you were using divination for that purposes then it's no issue. Harmless superstition is fine.
But things like personality tests and other pseudoscience see regular use in hiring and promotion. And that's just ridiculous, damaging for both "honest" applicants and the company, as such processes favour dishonest people.
That's the trick. It's about feelings of certainty, not actual measurable reproduceable predictions.
Most long-lived divination methods are very vague. Anything providing concrete predictions is easily proven wrong and discredited, only the vague survives.
But people rarely take ambiguous answers for what they are, and instead interpret them into something more certain.
And this lets divination exploit all kinds of biases. On top of the regular old confirmation bias, whenever the interpretation turns out wrong, people don't write off the divination method, but assume they merely "interpreted it wrong" (and often, the vagueness means they can retcon an interpretation that is true), and worse yet, assume that now they're better at interpreting so next time it's going to be a correct prediction.
Observe how little the personality tests actually say, they're just as ambiguous.
You're assuming here that there has to be "real" value at the root. This isn't really true.
Astrology, Tarot, the I Ching, or any other kind of divination all serve the same purpose: To provide certainty where there is none. To measure the unknowable.
People fear the unknown and risk, divination lets them feel like they have some certainty about the future.
Myers Briggs, DISC, and all the other "personality tests" are the same thing, for contemporary times.
They provide no actual measurement of applicants, Myers-Briggs is especially easy to cheat and dubious in science.
The benefit is that managers feel like they're taking less risk when hiring, but that is mere delusion.
A transformer can only memorize, it doesn't learn to do.
For what that concerns us here: LLMs will never learn to fact-check anything. They'll blindly regurgitate the facts they have been "taught", but never consider or evaluate "the paper cited for this fact on wikipedia is a bunch of bullshit".
Any attempt to use them to produce "facts" is ultimately just folly, in the same way Google's attempt to do so with it's search engine index is.
your rules will have not objective justification, and will be based on the personal philosophies of those in charge.
The "objective" safety standards are often a lot more philosophical than you think. There is no "objective" truth for road design, it is a trade-off between how important you deem the safety of pedestrians and cyclists, versus the convenience and throughput of cars.
But also, just look at fields like journalism. Journalistic ethics exist because without them they kill people.
Not ratting out your anonymous sources isn't some technical requirement laid down in the physics of the universe, it's a philosophical belief.
And yes. Choosing to not take people's personal data is a philosophical belief. But the harm isn't philosophical.
For personal ethics, they are mere opinion as you get to choose what those ethics are. For professional ethics, those ethics are the opinions of the relevant professional associations and regulatory bodies.
They are facts in the sense that "The Bar believes that it is unethical to lie to the court" is a fact. It is simply factually true that the legal profession holds that ethics belief.
And thus the point, anyone who seeks to join such a profession has to accept their ethics "as if" they were facts. You can't choose other professional ethics in the sense that you can choose to hold a different opinion.
"Living History" is a well crafted written experience, not procedurally generated slop.
The issue here is that LLMs can only act in-character if the world has already been built and written, if the prompts are so pre-chewed that you may as well just write the dialogue directly and get even better results.
Take Solaire of Astora. He's an interesting NPC not because of any depth of the dialogue, but because of how well in-tune he is to the world and game itself. A true believer in the old god, a beacon of optimism in a depressed dying world, and someone who sets the tone of the co-op multiplayer to be silly and fun.
But it isn't different. People have been using things like Markov chains to experiment with NPC dialogue for well over a decade.
It just never got widespread adoption because it's just not interesting, and LLMs are no different here. The dialogue is still empty, despite being deeper and more grammatically complex than previous attempts.
If every farmer in an RPG hands out the same "collect 20 bear asses" quest it doesn't matter if they all have "detailed" randomly generated backstories and can opine about the game world, real world philosophy, or the 2024 US elections.
It's not so much about formal logic, but general predictability.
even with formally composable languages like JavaScript, a semblance of unpredictability — akin to the "faerie logic" metaphor — still persists
And they're ridiculed for it, and as you state, we design around them or replace such systems entirely.
making continual interaction with them unavoidable
Technology is never unavoidable or "inevitable". We can choose not to use it, or when to use it.
The notion that we can pass a set of code or words through them once and expect a flawless result is simply illogical.
Yet that is what we expect when we put these systems into production use, especially when many proposed use cases are user-facing and subject to injection attacks.
Whether it be the writing of adcopy, the processing of loan applications, or generating code, mistakes in these tasks have very real consequences.
but you can tune your prompts so that 100% of the time they give you a valid result in the result
You can't though, that's the issue. Illustrative here are tokens like "SolidGoldMagikarp", but this does happen to "normal" sequences of tokens as well.
There is no filter you can build to keep out such mistakes, any set of otherwise normal tokens could trigger the model to produce wrong output.
Because of how large these models and most prompts are, even slight changes in things like attention can cascade into extremely different results.
there are definitely mechanisms and behaviors to discover that you can reason about.
It's faerie logic. The behaviours are mere trends and observations, not underlaying truth.
The faeries reward you for offering them fruit. But offer them apple which fell from the tree exactly 74 hours ago down to the second and they'll kill you. There is no way to know ahead of time which things will upset them.
The risk here is that you're fooled into believing these systems are understandable, that you know how they work, and that you'll mistakenly use them for something where the wrong results have consequences. You'll stop double-checking the output, all humans are lazy like that, and then you'll have disaster on your hands.
Prompt "engineering" is just writing prayers to forest faeries.
Whilst BASIC/JavaScript/etc are all magic incantations to a child, a child will soon figure out there's underlaying logic, and learn the ability to reason about what code does, and what certain changes will do.
With prompts, it's all faerie logic. There is nothing to learn, there are only magic incantations that change drastically if the model is updated.
Worse yet, the incantations cannot be composed. E.g. take the SQL statement "SELECT column FROM table WHERE column = [%s]". For any given string you insert here, the output is predictable. You can even know which characters would trigger an injection attack.
With prompts you cannot predict results. Any word, phrase, or sequence of characters may upset the faeries and cause the model to misbehave in who knows what way. No processing of user-input will stop injection attacks.
Whilst it's dubious to call current software development practices "engineering", it's utterly ridiculous to do so for prompt-writing.
> What a tricky problem to solve at scale because of the tragedy of the commons (As in; "I'm sure someone - not me - will be a sponsor for this project")
We have solutions for tragedies of the commons like this: Professional Associations, Unions, and Guilds.
They have the ability to raise funds through their membership fees, and can then spend that money on the common good. Membership can be 'encouraged' by requiring payment for commercial use if the company/it's employees aren't part of the association.
This solution is just not very popular, partly because of aversion to having to pay a % fee of your paycheque, partly because it's arguably "taking open source proprietary".
But if proper open-source is unsustainable, professional associations may yet be better than the alternatives of governments deciding which open source gets funding, or complete commercialisation of open source software.
Learn to take a hint. I'm not going to argue this on human terms because you're playing a dumb um-akshually game.
Computer reasoning systems can solve vastly more complex problems perfectly. Expert mathematicians can solve vastly more complex problems with only minimally increased errors. The ability of LLMs to solve reasoning problems completely disintegrates when the problems get more complex.
Trying to argue that LLMs are alike humans because of you can put these three into the buckets of "No mistakes" and "Some mistakes" is ridiculous.
Nobody is calling LLMs perfect reasoning machines.
Yes.
You said humans make mistakes, my point here is, humans make mistakes precisely because they stop doing reasoning and start doing blind pattern matching estimation of the answer.
The idea that you must make no mistake reasoning before you can be considered to be reasoning has no ground.
Reading comprehension.
I did not say no mistakes. I said that the failure pattern follows that of estimated guesses; Rapidly increasing errors as the size of the problem increases.
Whereas with computer reasoning, the rate of errors does not increase at all. And with (expert) humans the rate only goes up a little.
Did you even bother looking at the link?
You are missing the point.
I am not referring to literally English or any other language. I'm referring to the structure of language problems, which is vastly simpler than any moderately complex math or programming problem.
To more explicitly spell out the reason for my unimpressed-ness: They trained a pattern-repeating-machine and found that it will repeat some of their patterns, some of which were patterns trained on.
This does not demonstrate the ability to reason abstractly about new models, so I do not care.