Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.
> starting with an empirical result
What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.
Not Google search, but many companies are exploiting cheap labor over seas for different industrial use cases because our algos suck. Not just in manufacturing, there are tech companies outsourcing labeled data for tasks like object detection (Hive.ai as anvexample). Whether it makes you depressed or not, humans are cheaper than algorithms, and I don’t see that changing unless we abandon the current paradigm for ai/ml.
I think this is false. My general opinion of computer scientists and engineers (especially in silicon valley) lack the scientific training that one gets in other disciplines like in physics or other hard sciences. Psychology, linguistics, cognitive science and neuroscience has generated a rich diverse experimental data in the past 50 years, and now is the prime time for a theory to emerge to connect everything. To make what I’m saying clearer, you needed Newtonian mechanics, Maxwell and Plank and Coulomb to develop and discover the science of electricity and atoms in order for us to engineer the silicon transistor. Without the original scientific knowledge, building such devices would have been impossible. This machine learning, ai, deep learning and this obsession with benchmarks are, in my opinion, a hinderance to AGI
You’re making sweeping assertions that require domain experts in several different disciplines. The scientific evidence points to the contrary. It is demonstrated that humans have a general ability to learn wide range of things, without being genetically programmed to. None of us evolved to drive cars. Yet nearly everyone in my grandparents generation were able to learn this totally new skill despite being a new invention, where you couldn’t possibly have had time for evolution to act. They weren’t genetically evolved to drive, it was learned within their lifetime and generation. There are tribal humans in different parts of the world that haven’t developed written language. Yet you can teach them written language. Where’s your “pre trained brain” theory there?
Well, that’s not entirely true. Alpha go self played 29 million games of Go, which is in feasible for humans. Assuming a game of Go is 5mins, it would take a human 286 years to achieve the same, assuming this person doesn’t sleep, eat or do anything else other than just play Go. CPU time is an order of magnitude faster than real time, especially on GPU clusters.
I see BERT as non sensical. You need to be scientific and have a mathematical theory on how humans learn language, which is a multi disciplinary task requiring physicists, mathematicians, neuroscientists, cognitive psychologists and linguists. Benchmarks are useless, theories, models, experiments and testable predictions is how science progresses. You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked. The psychological, linguistic, evolutionary biology and neuroscience evidence doesn’t seem to corroborate. The evidence points stronger to humans having general learning and problem solving abilities. For instance, there was no evolutionary pressure for humans to be good at math or programming. I was not born knowing english or calculus or probability theory, these were learned abilities. Evolution favoured brain mechanisms that lead to behaviour for success in a rapidly changing world. Had I been born in ancient Rome as a farmer, I would learn to speak Latin, and learn how to be a successful farmer, instead of the physics, math, probability, computer, driving, reading skills that I learned in my life time.
My brain does nlp better than any system out there. I’m also able to ride bicycles and do motor control better than Boston Dynamics. I can also construct and prove math, do physics and code all of this in Matlab and C. My brain can handle all these wide range of tasks almost seamlessly with just 15 watts of power, that Silicon Valley’s super computers can barely do 1% of.
The wiki post mentioned that people are right 45% of the time. How does dl and ml stack up? Also not mentioned, how long does it take for humans to start recognizing new races, vs ml and dl?
I prefer Googles snippet to visiting a shitty site. This is one of the few instances that I fully support Google. If websites want to stay relevant, they need to be more inventive.
"No matter what the function, there is guaranteed to be a neural network so that for every possible input, x
, the value f(x)
or some close approximation) is output from the network"
Okay, so what? You require more and more neurons (ie. parameters) to approximate your function better and better. You can do the same with piecewise constant (Riemann sums). You can do this with trig functions too (Fourier transform).
"This result tells us that neural networks have a kind of universality."
I don't know what this statement means. What mathematical properties do neural networks have that other functions don't? The ability to approximate continuous functions isn't special. Given 5 points, I can perfectly fit an elephant to your function. And it's not like you are fitting the function with as few parameters as possible.
There is no "ai", it's the same bull crap we had in the 80s. We haven't had true innovation in this space for a very long time, the current hype was caused by GPU, cloud, and abundance of data for training.
This was a paper from Shazam from 2003. This is essentially what I proposed, there is no training. Shazam works pretty well. It's not even going into the mathematical consideration I went into.
>You'll never be able to develop features with the heuristic methods you described that will work as well as the features learned by a neural net.
The biggest issue I have is: "To do this we developed an entirely new system using convolutional neural networks to turn a few seconds of audio into a unique “fingerprint.”
Why did you pick a neural network? What mathematical properties does a neural network have that makes it appealing to this problem? How were the networks trained? Back propagation? It doesn't converge, and worse learning weights for a new batch can cause you to forget previous batches. This isn't a desirable property of neural networks or back propagation. You probably had a lot of heuristics on top, fine. How do you know that the weights you ended up with will always work in practise? Given an arbitrary track, you can encode it? What about growing the database? Does the neural network get updated for new songs, or do you use the same neural network to fingerprint new songs and update the data base?
Here's how I would have done it:
A song file is just a sequence of amplitudes. I would do some kind of an interpolation of piece-wise trig function. Trig functions have very desirable properties: they are continuous everywhere, and infinitely differentiable. Moreover, a sine basis decomposition will be able to reconstruct the original signal very well. This is great, because now you can use theories from DSP and fourier analysis. So we take the entire song, do a continuous time discrete cosine transform, in a block size of 32. Now you compute the square norm of all feature vectors, sort them, eliminate the vectors that are within 1e-3 radius (they are too similar to each other, there's not point in keeping them) and only store the top 25% of feature vectors by the square norm. The 25% cut off threshold and 1e-3 radius of similarity are heuristics, and adjustable parameters.
Now you have a database. For a new song, repeat the procedure, and get a feature vector for every 32 interval. There are probably theories in DSP you can use to get a better similarity measure, but for now, we'll just use the L2 norm of the difference. Do a nearest neighbour search in your data base for all feature vectors, and rank the results based on hits. I can run all of this on a computer from 2000s which are crappier than modern phones, and have the entire backend run on equally crappy hardware too. All parts of what I'm doing are fully deterministic, updating the DB is incredibly fast, CTDCT is super fast, there are no questions of convergence, no need for training. You can probably increase the accuracy and speed by doing some DSP and doing the nearest neighbour search based on different voice, bass, instrumental etc. features.
In practise how would it compare to your neural network? No idea, but I imagine it should be very competitive. The big benefits are that you have only 3 parameters (radius of similarity, cut off threshold and block size). This seems very easy to bench mark against, it should take like a week to implement. I'm not sure about the compression of the finger print however. Not sure how much space 1000000 songs will take (probably 25% since that was our cutoff). You can probably borrow psycho acoustics to make a better data base, and get a better compressed representation. Another alternative would be to down sample the song to 64kbps before hand.
>you do realize that NN/AI is totally state of the art for many tasks?
Being state of the art doesn't imply that these things will solve these problems. In ML terms, how do you know that NN/AI isn't a local maxima that we need to jump out of? All NLP systems are joke. Sure replace Watson with DL, might perform better on Jeopardy. But in real conversations? Forget it.
I wouldn't bet on these things. NN will win, but not the back propagation, ReLu, sigmoid or whatever pseudo science that is the current buzzword. There is 50 years worth of understanding in actual neuroscience and cognitive modelling that no one has paid attention to, and new design principles are emerging that will influence mathematics.
>Some of the most brilliant people in the world have been working on image recognition, voice recognition etc. and AI is crushing all of their work.
This is very true. I take my stronger statements back, MAINSTREAM mathematicians attempting this problem are all wrong, and have been wrong for 50 years. But you do need the right theory, and the right math that realizes this theory.
"AI" is superficially beating the work in computer vision. Computer vision is complete bogus. The gabor filters, fourier transfroms etc. are all wrong conceptually. The known methods do abysmally on basic tasks like object recognition, texture segmentation etc. But they keep trying it.
I would take this one step further: computer vision, audio and NLP researchers have been stuck in a rut for the past 50 years. DL is beating THEIR math, but this is because of data and computation speed, not because of any insights. But DL is also wrong, and giving you an illusion of progress. Both of these things are doomed to go the way of GOFAI.
I can go into great detail and carefully explain why MAINSTREAM contemporary ideas in math for vision, audition and language are completely wrong, and have been wrong for 50 years. What is the right model? Like I mentioned before, the right ideas are emerging, neural networks will dominate, just not DL.
Machine learning is glorified curve fitting. DL isn't even mathematically sound, back propagation has no proof of convergence. Quantitative math is about extracting natural laws, and mapping them to mathematical structures. You could use DL to predict planetary motion, and get pretty good at it. But this isn't a quantitative understanding of the world. You didn't learn anything. Physics in contrast has the laws of motion and gravitation. You can directly model arbitrary planets. Moreover, you can model arbitrary rigid bodies, from cars to space shuttles. Your ML, DL random forrest etc. all use math, sure. But so did the Keplarian models of motion. You aren't qualitatively deducing math that governs the world, but forcing an arbitrarily chosen mathematical structure to your data.
Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.
> starting with an empirical result
What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.