> Schmidhuber is cited far more often than he should be
You haven't even read the paper, have you? Otherwise you'd see that it's Hinton and Bengio who are cited far more often than they should be. Just look at disputes B1, B2, B5, H2, H4, and H5 to see how they republished parts of his work again and again without citing it. No honest scientist can approve of something like that.
Agreed. The piece anticipated this straw man argument:
> "the inventor of an important method should get credit for inventing it. She may not always be the one who popularizes it. Then the popularizer should get credit for popularizing it (but not for inventing it)." Nothing more or less than the standard elementary principles of scientific credit assignment.[T22] LBH, however, apparently aren't satisfied with credit for popularising the inventions of others; they also want the inventor's credit.[LEC]
In the past few hours I have had more time to look at the entire piece and download some of the referenced papers. So far I haven't found any claim that's factually inaccurate.
I think there is a reason why the ACM Turing awardees have never tried to defend themselves by presenting facts to the contrary: because they can't.
This might get interesting:
> The "Policy for Honors Conferred by ACM"[ACM23] mentions that ACM "retains the right to revoke an Honor previously granted if ACM determines that it is in the best interests of the field to do so." So I ask ACM to evaluate the presented evidence and decide about further actions.
Are you talking about the 1985 Genetic Programming paper by Cramer? Unlike Hinton and Bengio, Schmidhuber has corrected himself:
> BTW, I committed a similar error in 1987 when I published what I thought was the first paper on Genetic Programming (GP), that is, on automatically evolving computer programs[GP1][GP] (authors in alphabetic order). At least our 1987 paper[GP1] seems to be the first on GP for codes with loops and codes of variable size, and the first on GP implemented in a Logic Programming language. Only later I found out that Nichael Cramer had published GP already in 1985[GP0] (and that Stephen F. Smith had proposed a related approach as part of a larger system[GPA] in 1980). Since then I have been trying to do the right thing and correctly attribute credit.
I can see some angry comments here, but so far I have not seen any facts that refute his claims. Once I spent a long time reviewing a related paper on Hacker News, and I think he is right about disputes B1, B2, B5, H2, H4, H5. I'd have to study the others more closely:
B: Priority disputes with Dr. Bengio (original date v Bengio's date):
B1: Generative adversarial networks or GANs (1990 v 2014)
B2: Vanishing gradient problem (1991 v 1994)
B3: Metalearning (1987 v 1991)
B4: Learning soft attention (1991-93 v 2014) for Transformers etc.
B5: Gated recurrent units (2000 v 2014)
B6: Auto-regressive neural nets for density estimation (1995 v 1999)
B7: Time scale hierarchy in neural nets (1991 v 1995)
H: Priority disputes with Dr. Hinton (original date v Hinton's date):
H1: Unsupervised/self-supervised pre-training for deep learning (1991 v 2006)
H2: Distilling one neural net into another neural net (1991 v 2015)
H3: Learning sequential attention with neural nets (1990 v 2010)
H4: NNs program NNs: fast weight programmers (1991 v 2016) and linear Transformers
H5: Speech recognition through deep learning (2007 v 2012)
H6: Biologically plausible forward-only deep learning (1989, 1990, 2021 v 2022)
L: Priority disputes with Dr. LeCun (original date v LeCun's date):
L1: Differentiable architectures / intrinsic motivation (1990 v 2022)
L2: Multiple levels of abstraction and time scales (1990-91 v 2022)
L3: Informative yet predictable representations (1997 v 2022)
L4: Learning to act largely by observation (2015 v 2022)
As the author points out: there isn’t any one Kalman filter. The Kalman filter is really a recipe for constructing optimal linear predictive filters, but the actual characteristics of the resulting filter will depend on the dynamics, state variables, and sensors that you’ve tuned it for, and that dependence gets reflected numerically in the various matrices that your specific Kalman filter is built from.
LeCun claims four "main original contributions" and Schmidhuber basically debunks them one by one, for example:
> (IV) your predictive differentiable models "for hierarchical planning under uncertainty" - you write: "One question that is left unanswered is how the configurator can learn to decompose a complex task into a sequence of subgoals that can individually be accomplished by the agent. I shall leave this question open for future investigation."
> Far from a future investigation, I published exactly this over 3 decades ago: a controller NN gets extra command inputs of the form (start, goal). An evaluator NN learns to predict the expected costs of going from start to goal. A differentiable (R)NN-based subgoal generator also sees (start, goal), and uses (copies of) the evaluator NN to learn by gradient descent a sequence of cost-minimizing intermediate subgoals [HRL1].
A good way of ending this article: "I was fascinated by the way in which the personal agendas of committee members were clothed in seemingly reasonable attempts to place restrictions on the prize," says Christopher Hollings, a historian of mathematics at the University of Oxford in the United Kingdom, who attended Barany's talk. "It is a nice and interesting reminder that mathematicians are people, too."
> The key result is not the introduction of ReLU, this is a misdirection. The key result is the outstanding performance on a general image data set by Alexnet. If the predecessors did all of the work, why was Hinton's lab the first to produce these results.
When Fukushima published ReLUs in 1969 and CNNs in 1979, there were neither decent computers nor competitions. No excuse for not citing him.
> ReLU is an absurdly simple gate. The question revolved around its effectiveness, which was proven by Hintons lab.
Many good things are simple. They should have cited the creator, no matter how much they profited later from faster computers or novel datasets or the like.
> The key result is the outstanding performance on imagenet. If Schmidhuber was the actual pioneer, why wasn't he able produce the same results before Hinton?
Did his team ever participate in imagenet? Apparently not. He writes about DanNet: "For a while, it enjoyed a monopoly. From 2011 to 2012 it won every contest it entered, winning four of them in a row (15 May 2011, 6 Aug 2011, 1 Mar 2012, 10 Sep 2012)"
> NN were known to work for hand writing recognition since the 90s (including papers by Hinton). Dannet being able to do it for Chinese characters in 2010s is unremarkable.
The remarkable thing is that "DanNet was the first pure deep CNN to win computer vision contests." Before DanNet, other methods won the competitions. DanNet changed that.
However, the CNN pioneer was Fukushima who introduced the CNN architecture and ReLUs. Hinton did not cite him.
> You are well aware that not citing an earlier paper with different implementation and results is not plagiarism. There is absolutely no evidence of plagiarism anywhere.
So what exactly constitutes plagiarism? It's not about good or bad faith, it's about checking who did it first. If you are using building blocks from previous papers, you must cite them. Schmidhuber cites the difference between unintentional [PLAG1] and intentional plagiarism [FAKE2]:
[PLAG1] Oxford's guidance to types of plagiarism (2021). Quote: "Plagiarism may be intentional or reckless, or unintentional."
[FAKE2] L. Stenflo. Intelligent plagiarists are the most dangerous. Nature, vol. 427, p. 777 (Feb 2004). Quote: "What is worse, in my opinion, ..., are cases where scientists rewrite previous findings in different words, purposely hiding the sources of their ideas, and then during subsequent years forcefully claim that they have discovered new phenomena."
More quotes: "If one "re-invents" something that was already known, and only becomes aware of it later, one must at least clarify it later, and correctly give credit in follow-up papers and presentations." ... "And the authors did not cite the prior art - not even in later surveys."
This is crucial. Even later they did not cite the original sources.
> Following up on your logic is absurd, because I can conveniently state that back prop is just the chain rule in differentiation by Newton and everyone else has plagiarized from him.
The paper apparently both anticipated and corrected your claim (it wasn't Newton): "Some claim that "backpropagation is just the chain rule of Leibniz (1676) & L'Hopital (1696)." No, it is the efficient way of applying the chain rule to big networks with differentiable nodes (there are also many inefficient ways of doing this). It was not published until 1970.[BP1]"
[BP1] S. Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis (in Finnish), Univ. Helsinki, 1970. See chapters 6-7 and FORTRAN code on pages 58-60. PDF. See also BIT 16, 146-160, 1976. Link. The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation.
> And ReLU was plagiarized by Fukushima from neuroscience researchers.
Really? Can you prove this? Do you have a reference?
Quote from the article: "Dr. Taylor’s standard procedure included administering laxatives and bloodletting from the patient. In addition, Taylor used the blood from slaughtered pigeons as eye drops, and often applied a baked apple to the eye with a bandage. Adding to the abuse, he charged large sums for his procedures—especially if he judged that the patient was wealthy. Bach survived a few weeks after the second procedure, but it seems likely that these failed eye operations directly caused his rapid decline and death. Taylor’s approach was extremely unhygienic, and thus likely to lead to post-surgical infections. I remind readers that antibiotics didn’t exist back then, and an infection, once it had set in, was often fatal. In any event, Bach died on July 28, 1750 at age 65."
You conveniently left out the plagiarism part regarding ReLUs:
> 8. LBH devote an extra section to rectified linear units (ReLUs), citing papers of the 2000s by Hinton and his former students, without citing Fukushima who introduced ReLUs in 1969[RELU1-2] (see Sec. XIV).
This is only one of many concrete examples given.
DanNet obviously worked on all kinds of image data, otherwise it would not have won all those competitions before the similar AlexNet. However, the CNN pioneer was Fukushima who introduced CNNs and ReLUs.
6th paragraph: Someone should have said this to me: Imagine a flashy spaceship lands in your backyard. The door opens and you are invited to investigate everything to see what you can learn. The technology is clearly millions of years beyond what we can make. This is biology. –Bert Hubert, “Our Amazing Immune System”