Andrew Ng: Unbiggen AI(spectrum.ieee.org)
spectrum.ieee.org
Andrew Ng: Unbiggen AI
https://spectrum.ieee.org/andrew-ng-data-centric-ai
89 comments
I was going to interview at LandingAI. I was asked before the interview to install a spyware browser extension to monitor my traffic to detect if I was cheating during the interview. I respectfully declined and didn't have that interview.
Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
> Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
Not necessarily, at least on the first point. Someone could be getting coached.
A few years ago, a coworker of mine hired a contractor onto his team and was convinced the person who actually showed up was not the person who he interviewed (over the phone). He also thought the guy who did show up was getting a lot of help day-to-day from somewhere. The guy was a contractor, so it wasn't a huge problem because we could drop him quickly, but I would have never expected someone would do anything like that. However, it kind of makes sense as a scam: be a decent developer, get a stable of unhirable incompetents, and rotate them through companies while taking a cut of their salary.
Not necessarily, at least on the first point. Someone could be getting coached.
A few years ago, a coworker of mine hired a contractor onto his team and was convinced the person who actually showed up was not the person who he interviewed (over the phone). He also thought the guy who did show up was getting a lot of help day-to-day from somewhere. The guy was a contractor, so it wasn't a huge problem because we could drop him quickly, but I would have never expected someone would do anything like that. However, it kind of makes sense as a scam: be a decent developer, get a stable of unhirable incompetents, and rotate them through companies while taking a cut of their salary.
You cannot really prevent those kinds of cheats. Even if you use the most insidious spyware a coach can advice the interviewee from a different device.
The only way to prevent those kind of scams is to put all employees in probation for the first months of work and fire them if they don't perform, like it's common in the UK.
The only way to prevent those kind of scams is to put all employees in probation for the first months of work and fire them if they don't perform, like it's common in the UK.
Agreed. Whenever I hear stories like this, I recall this recent essay [1] that said it quite well -
> designing a human process around pathological cases leads to processes that are themselves pathological
[1] - https://jacobian.org/2022/feb/14/that-wild-aam-story/
> designing a human process around pathological cases leads to processes that are themselves pathological
[1] - https://jacobian.org/2022/feb/14/that-wild-aam-story/
I once heard where a dev literally offshored his work and had entire teams working on his tasks. He was employed by multiple companies at once and paid a small fraction of his combined pay to offshore team.
Eventually he got caught trying to manage all this
Eventually he got caught trying to manage all this
i mean, people are good at finding clever ways to cheat.
Well, Ng is also one of those people who believe that we should all work 70+ hours per week:
https://news.ycombinator.com/item?id=15251769
https://news.ycombinator.com/item?id=15251769
~80hrs on topic A squeezes the available time for being acquainted with the rest. [Edited because there was little way not to make the former formulation read, unwillingly, nasty]
Some of us believe instead on the advantage of being a polymath, (also) to be able to export wisdom from other contexts into the current work.
Also in terms of the proper ground to facilitate innovation.
Some of us believe instead on the advantage of being a polymath, (also) to be able to export wisdom from other contexts into the current work.
Also in terms of the proper ground to facilitate innovation.
Musk recommends 80-100hr weeks, every week
Source: https://www.youtube.com/watch?v=GtaxU6DZvLs&t=1m20s
Source: https://www.youtube.com/watch?v=GtaxU6DZvLs&t=1m20s
Jack Ma is a slacker...Only 72h per week.
"Jack Ma says his employees should work 12 hours a day, 6 days a week" https://twitter.com/i/events/1116787491707731968?lang=en
"Jack Ma says his employees should work 12 hours a day, 6 days a week" https://twitter.com/i/events/1116787491707731968?lang=en
Maybe that's why CCP took the company from him? Not enough totalitarian work conditions?
It's literally our job to not just assume the possible solution that rolls off the top of our heads might not be the most up to date / best practice and to research it
Agreed. A decent interviewer can also determine a person’s understanding of a topic by simply talking to them about it. IE why did you build a model like this? What diagnostics did you use? Have you tried ____ before in your career?
I'd just note that if pushed by circumstances (if one was willing to be interviewed in spite of their ways), the interview environment could be (would be) on a throwaway virtual machine...
Possibility which, by the way, makes the interviewer's cautionary move generally useless.
Possibility which, by the way, makes the interviewer's cautionary move generally useless.
Or, in 2022, one could reach into their pocket just use a phone, making the interviewer's cautionary move generally useless.
> just use a phone
I assumed that keeping looking in the direction of the camera was relevant (in their idea).
I assumed that keeping looking in the direction of the camera was relevant (in their idea).
It's fairly easy to augment a video stream to paste on eyes that always look in the direction of the camera. It's the digital equivalent of glasses with eyes on them[1].
1. https://www.amazon.com/glasses-eyes-them/s?k=glasses+with+ey...
1. https://www.amazon.com/glasses-eyes-them/s?k=glasses+with+ey...
I have multiple monitors
...Which one supposedly would not consult during an interview.
No matter how you arranged them. And I would never use a USB camera between them, either.
I suppose if you’re clever enough to set up a VM in order to evade detection, that’s a pretty positive aptitude signal in its own right (though pretty negative on the behavioral/ethics side).
> if you’re clever enough to set up a VM in order to evade detection
It's what you would do anyway, unless you suppose that the interviewee would ever install dubious software on his core machine.
It's what you would do anyway, unless you suppose that the interviewee would ever install dubious software on his core machine.
Missed opportunity to say you landed an interview at LandingAI :-)
My understanding is that they are trying to automate the data preparation steps that seasoned ML practitioners are doing anyway today.
The fact that he tries this in manufacturing makes the case stronger. In most manufacturing companies you do not have access to top ML talent.
You have Greg who knows python and recently visualized some production metrics.
If we could empower Greg with automated ML libraries that guide him in the data preparation steps in combination with precooked networks like autogluon, then manufacturing could become a huge beneficiary of the ML revolution.
The fact that he tries this in manufacturing makes the case stronger. In most manufacturing companies you do not have access to top ML talent.
You have Greg who knows python and recently visualized some production metrics.
If we could empower Greg with automated ML libraries that guide him in the data preparation steps in combination with precooked networks like autogluon, then manufacturing could become a huge beneficiary of the ML revolution.
Greg probably also knows SAS and AMPL, and has a good knowledge of ops research, which is within stone-tossing distance of whatever ML is pretending to be this week.
After 15 years of experience with SAS this sounds to me like saying "knowing how to write and having a pen makes you to a poet". But it depends on how far you can toss a stone...
OR and ML have their own space in manufacturing.
OR is perfect when you can describe explicitly what the decision space is and what the restrictions are.
ML is great fit when you want to identify and use patterns. Quality control with machine vision is a good application for ML. NLP for PDF documents is a huge field for manufacturing as well. Companies have so much data in email attachments that they do not currently take advantage of.
OR is perfect when you can describe explicitly what the decision space is and what the restrictions are.
ML is great fit when you want to identify and use patterns. Quality control with machine vision is a good application for ML. NLP for PDF documents is a huge field for manufacturing as well. Companies have so much data in email attachments that they do not currently take advantage of.
> OR is perfect when you can describe explicitly what the decision space is and what the restrictions are.
As opposed to having to figure it out later from the outputs of a black box?
> Quality control with machine vision is a good application for ML.
I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
> NLP for PDF documents is a huge field for manufacturing as well.
NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
As opposed to having to figure it out later from the outputs of a black box?
> Quality control with machine vision is a good application for ML.
I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
> NLP for PDF documents is a huge field for manufacturing as well.
NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
> As opposed to having to figure it out later from the outputs of a black box?
Not all problems can be formulated as a set of explicit equalities, constraints and variables (e.g. machine vision). If explicit modeling is an option, of course you should do it. I am seeing efforts to try reinforcement learning on systems that we know how to describe with equations, and of course the results are laughable compared to the traditional methods.
> I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
In one big manufacturing company they were using Machine vision and a cheap web camera to control flaring. Could they do it with fancy sensors instead? Of course, but it would be more expensive, and they never did in the past.
Another manufacturing company is using machine vision to raise an alarm if the door of a cargo car of a train is not closed after loading. Could they install sensors in all of the doors of the train instead? Sure, but it would be cost prohibitive.
>NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
In manufacturing we have multiple people opening pdfs from emails to copy contract numbers to excel spreadsheets. Others are getting orders in emails and then type them in SAP manually. I think that these tasks can be automated specially with the recent versions of NLP networks.
>I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
All is good :) There has been a lot of unsubstantiated hype in ML, made even worse by big consulting companies and cloud providers who just sell the hype.
Not all problems can be formulated as a set of explicit equalities, constraints and variables (e.g. machine vision). If explicit modeling is an option, of course you should do it. I am seeing efforts to try reinforcement learning on systems that we know how to describe with equations, and of course the results are laughable compared to the traditional methods.
> I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
In one big manufacturing company they were using Machine vision and a cheap web camera to control flaring. Could they do it with fancy sensors instead? Of course, but it would be more expensive, and they never did in the past.
Another manufacturing company is using machine vision to raise an alarm if the door of a cargo car of a train is not closed after loading. Could they install sensors in all of the doors of the train instead? Sure, but it would be cost prohibitive.
>NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
In manufacturing we have multiple people opening pdfs from emails to copy contract numbers to excel spreadsheets. Others are getting orders in emails and then type them in SAP manually. I think that these tasks can be automated specially with the recent versions of NLP networks.
>I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
All is good :) There has been a lot of unsubstantiated hype in ML, made even worse by big consulting companies and cloud providers who just sell the hype.
There is a significant amount of research from the field of computer vision before ML even existed. It was quite robust as well within certain constraints. Those techniques simply did not generalize anywhere even close to as well as deep learning.
However, that said, in a tightly controlled environment such as a manufacturing line trying to spot defects I would imagine they would have a good chance at performing a lot better than deep learning.
A lot of the advancements in deep learning have also come out of ideas from that research. While they didn't use the techniques directly, there is a lot of knowledge that we'd be lost without.
This is one thing that scares me about ML. We are losing research into the fundamental physics/science to deeply understand these things and instead just throwing models at them.
However, that said, in a tightly controlled environment such as a manufacturing line trying to spot defects I would imagine they would have a good chance at performing a lot better than deep learning.
A lot of the advancements in deep learning have also come out of ideas from that research. While they didn't use the techniques directly, there is a lot of knowledge that we'd be lost without.
This is one thing that scares me about ML. We are losing research into the fundamental physics/science to deeply understand these things and instead just throwing models at them.
[deleted]
I work in this space: most "cool" ML is useless, and stakeholder are very skeptical of new modelling techniques. It is a long slog of EDA and finding actionable causality. Deep learning, modern reinforcement learning... are not the best fit here.
However I have seen CV and NLP useful here and there... but it is not the bread and butter.
However I have seen CV and NLP useful here and there... but it is not the bread and butter.
A tangent, if you have time: where would I go for a primer on operations research and/or discrete event simulation?
My thought is that Goldratt's "The Goal" / theory of constraints is a useful way of thinking about optimizing throughput in a computer system. http://www.qdpma.com/Arch_files/RWT_Nehalem-5.gif plus an instruction latency table is something like a well modeled factory. (The Phoenix Project applies these principles to project management, which I think is a somewhat less useful analogy!)
I'm curious about applying existing tools to modeling things like: how will this multi-tiered application behave when it gets a thundering herd of requests? What if I tweak these timeouts, adjust this queue, make a particular system process requests on a last-in-first-out basis? Can I get a pretty visualization of what would happen?
My thought is that Goldratt's "The Goal" / theory of constraints is a useful way of thinking about optimizing throughput in a computer system. http://www.qdpma.com/Arch_files/RWT_Nehalem-5.gif plus an instruction latency table is something like a well modeled factory. (The Phoenix Project applies these principles to project management, which I think is a somewhat less useful analogy!)
I'm curious about applying existing tools to modeling things like: how will this multi-tiered application behave when it gets a thundering herd of requests? What if I tweak these timeouts, adjust this queue, make a particular system process requests on a last-in-first-out basis? Can I get a pretty visualization of what would happen?
lol-ing at "Whatever ml is pretending to be this week"
so funny, because so accurate :)
so funny, because so accurate :)
Visual inspection in manufacturing is a very solved problem, especially in the AI field. The big bucks are in pattern matching anyway... it's a dumb company.
That is the problem with generalization and cop outs like these. It's no good to people in the field doing actual work where the devil is in the detail.
Big data is fairly important to a lot of things, for example I was listening to Tesla's use of Deep net models where they mentioned that there were literally so many variations of Stop Signs that they needed to learn what was really in the "tail" of the distribution of Stop Sign types to construct reliable AI
Big data is fairly important to a lot of things, for example I was listening to Tesla's use of Deep net models where they mentioned that there were literally so many variations of Stop Signs that they needed to learn what was really in the "tail" of the distribution of Stop Sign types to construct reliable AI
Interestingly, when you learn how to drive you need to see approximately one example and you're able to identify them all.
That is called transfer learning. You might only need to see one photo of a sign to identify it in real life (although arguably learner drivers take a while to notice signs) but that is only because you have been training on identifying generic objects since you left the womb.
You brain already knows how to select the most important features of a sign. The shape, the size and the color. You have also learned how to understand the text on the sign.
A new born baby does not have that ability.
This is applied in ANN as well. Transfer learning is using a pre-trained neural network, which has already learned identifying objects, and then using it to train on identifying a new, usually smaller, set of objects using, usually, a lot less training data. That is what Andrew is talking about in the article.
You brain already knows how to select the most important features of a sign. The shape, the size and the color. You have also learned how to understand the text on the sign.
A new born baby does not have that ability.
This is applied in ANN as well. Transfer learning is using a pre-trained neural network, which has already learned identifying objects, and then using it to train on identifying a new, usually smaller, set of objects using, usually, a lot less training data. That is what Andrew is talking about in the article.
> The shape, the size and the color.
And the context. For example, self-driving cars need to account for "Pizza Stop" restaurant signage, placards stuck to telephone poles that say things like "Stop Cancer", stop signs retracted into the sides of school buses, signs with additional instructions like "Stop when lights flashing", road workers with handheld stop signs, and the unconventional stop signs you see in parking lots.
You can probably get pretty far by checking the proximity to the road, height, dimensions, orientation, what it's mounted on, and if the sign incorporates any other text. But you can't just scan some pixels for "red octagon with STOP on it".
And the context. For example, self-driving cars need to account for "Pizza Stop" restaurant signage, placards stuck to telephone poles that say things like "Stop Cancer", stop signs retracted into the sides of school buses, signs with additional instructions like "Stop when lights flashing", road workers with handheld stop signs, and the unconventional stop signs you see in parking lots.
You can probably get pretty far by checking the proximity to the road, height, dimensions, orientation, what it's mounted on, and if the sign incorporates any other text. But you can't just scan some pixels for "red octagon with STOP on it".
> because you have been training on identifying generic objects since you left the womb.
We can go back even further - your genes carry information about the structure and function of your brain and this has been refined by natural selection over the course of human evolution. Humans don't start from scratch with randomly initialised weights.
We can go back even further - your genes carry information about the structure and function of your brain and this has been refined by natural selection over the course of human evolution. Humans don't start from scratch with randomly initialised weights.
I'm not sure "training" is the right way to think about it. Children don't train to identify objects, they quickly develop the ability to recognize objects and are able to correlate them with prior information that was retained and learned. Case in point: if you take a child born blind, give them the ability to see, they are immediately able to recognize and correlate objects around them.
> Children don't train to identify objects
I have to disagree. They spend an inordinate amount of time trying to understand what they see, taste and hear.
> able to correlate them with prior information that was retained and learned
This is what we call inference.
> Case in point: if you take a child born blind, give them the ability to see, they are immediately able to recognize and correlate objects around them.
Not everyone who is legally blind can see absolutely nothing, but people who have recovered from complete vision loss [1] have problems. Mike May [2] lost vision as 3 year old child and regained it in his 40s. Despite seeing for the first three years of his life, years after regaining vision he was unable to see in 3D or recognize people from faces alone.
Blind people do not lack spatial awareness, so being able to recognize objects with context if they regained sight with would not surprise me. There are blind people that can "see" with echo location using parts of the brain associated with visual processing [3] But for example in Mike's case, he was unable to recognize close family by their faces years after regaining vision, he needed additional context.
Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
[1] https://en.wikipedia.org/wiki/Recovery_from_blindness
[2] https://en.wikipedia.org/wiki/Mike_May_(skier)
[3] https://en.wikipedia.org/wiki/Human_echolocation
I have to disagree. They spend an inordinate amount of time trying to understand what they see, taste and hear.
> able to correlate them with prior information that was retained and learned
This is what we call inference.
> Case in point: if you take a child born blind, give them the ability to see, they are immediately able to recognize and correlate objects around them.
Not everyone who is legally blind can see absolutely nothing, but people who have recovered from complete vision loss [1] have problems. Mike May [2] lost vision as 3 year old child and regained it in his 40s. Despite seeing for the first three years of his life, years after regaining vision he was unable to see in 3D or recognize people from faces alone.
Blind people do not lack spatial awareness, so being able to recognize objects with context if they regained sight with would not surprise me. There are blind people that can "see" with echo location using parts of the brain associated with visual processing [3] But for example in Mike's case, he was unable to recognize close family by their faces years after regaining vision, he needed additional context.
Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
[1] https://en.wikipedia.org/wiki/Recovery_from_blindness
[2] https://en.wikipedia.org/wiki/Mike_May_(skier)
[3] https://en.wikipedia.org/wiki/Human_echolocation
> I have to disagree. They spend an inordinate amount of time trying to understand what they see, taste and hear.
To understand yes, however I meant "identify" in the sense of recognizing an object. I show a picture of an apple to a child a bit over a year old, hide it, put it in a basket of other things and present it back to them, they will be able to identify it. Or if you show an object to a child, then hide the object, they can realize that the object is no longer there, regardless of their understanding of what happened to the object.
> Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
I don't think it's that simple, I would say it's both. I don't question learning plays a big role in recognition, just pointing out that a large amount intrinsic knowledge also exists from early child development. Many of those cases where someone regains eyesight happens much later in life, at a time when their brains have largely matured to the point that neuroplasticity is pretty much over for them. Having someone's brain develop with almost no visual input at a young age is bound to mean that their visual cortex and its connections to everything else doesn't develop as it should.
From a quick search, it appears that to some degree children born categorically blind can recover all the way up to teenage life. [1] But indeed it's likely less effective than a younger child undergoing a similar procedure (which I don't think is really that rare: it's hard to diagnose vision problems at young age, and a lot of children who get necessary corrective surgery at young age turn out fine).
[1] https://www.nature.com/articles/nature.2014.14592
To understand yes, however I meant "identify" in the sense of recognizing an object. I show a picture of an apple to a child a bit over a year old, hide it, put it in a basket of other things and present it back to them, they will be able to identify it. Or if you show an object to a child, then hide the object, they can realize that the object is no longer there, regardless of their understanding of what happened to the object.
> Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
I don't think it's that simple, I would say it's both. I don't question learning plays a big role in recognition, just pointing out that a large amount intrinsic knowledge also exists from early child development. Many of those cases where someone regains eyesight happens much later in life, at a time when their brains have largely matured to the point that neuroplasticity is pretty much over for them. Having someone's brain develop with almost no visual input at a young age is bound to mean that their visual cortex and its connections to everything else doesn't develop as it should.
From a quick search, it appears that to some degree children born categorically blind can recover all the way up to teenage life. [1] But indeed it's likely less effective than a younger child undergoing a similar procedure (which I don't think is really that rare: it's hard to diagnose vision problems at young age, and a lot of children who get necessary corrective surgery at young age turn out fine).
[1] https://www.nature.com/articles/nature.2014.14592
Is there some underlying point to this statement? It comes off as a passive dismissal of something but I'm not sure what. It might be helpful to directly state what you're trying to say so that other people can engage with it.
It doesn't seem like the other replies are having trouble engaging with it. Since we're giving each other advice, you should use your down votes instead of pontificating that other people's comments are "passive dismissals" when you don't like or understand them.
It does feel though that a model like the human mind will be very fundamentally different from any of the models of today. No?
Like the NN State of the art models of today are so different from state of the art 12 or so years ago which was SVMs.
Like the NN State of the art models of today are so different from state of the art 12 or so years ago which was SVMs.
Your brain is also the result of billions of years of evolutionary "training." Neural nets start from scratch.
when you learn to drive you need [approximately 16 years training your visual system hardware that took 6 million years to evolve and] to see approximately one example and you're able to identify them all.
FTFY.
Yet Tesla have been working on both the hardware and software for 10 years? Amazing progress right?
FTFY.
Yet Tesla have been working on both the hardware and software for 10 years? Amazing progress right?
[deleted]
If you’re going back 6 million years on evolution, would it make sense to push back the start date for Tesla’s work as well?
It seems the right parallel to evolution might include predecessor inventions in vision, computation, and beyond.
It seems the right parallel to evolution might include predecessor inventions in vision, computation, and beyond.
Sure… so 100 years? 2000? Take your pick. AI Vision is developing ridiculously fast.
Sounds like they missed the forest and instead "deep learned" all the variations of trees.
Do you have a link for that Tesla talk?
I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think it would be interesting to imagine a point of diminishing return on additional data if we consider that our ability to perceive is probably largely based on two parts - sensory input and knowledge. Note that I’m making an explicit distinction here on the difference between data and knowledge.
For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded.
This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things.
Somewhat related, we have a saying in Korean – ‘you see as much as you know’.
For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded.
This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things.
Somewhat related, we have a saying in Korean – ‘you see as much as you know’.
> more data equals
It does in general, but what is elaborated and how? Structuring patterns is not the same as "knowledge" (there are missing subsystems), and that fed data is not fed efficiently, with ideal efficiency - compare with the realm in which "told one notion you acquire it" (this while CS is one of the disciplines focusing on optimization, so it would be a crucial point).
It does in general, but what is elaborated and how? Structuring patterns is not the same as "knowledge" (there are missing subsystems), and that fed data is not fed efficiently, with ideal efficiency - compare with the realm in which "told one notion you acquire it" (this while CS is one of the disciplines focusing on optimization, so it would be a crucial point).
I have a feeling that too much knowledge might slow learning process as it's harder to spot/test observe steepest gradient. At least that's how it feels intuitively from human PoC. From computation that would be just little more computation but I guess would mean slower convergence also. Taking math as more extreme example it's hard to understand something complex unless you understand basic algebra.
Anyone knows if this might be true mathematically speaking? Does order of data matters?
Anyone knows if this might be true mathematically speaking? Does order of data matters?
Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.
Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone.
Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions.
Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions.
Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
Yep. Perhaps what I should have written is that much our knowledge is tacit in nature. We implicitly understand that things we can sense are limited projections of the real world and are able to derive mental model of what that reality might be more or less. Chomsky also talks about this notion of separating mind and body as an incorrect approach to understanding human intelligence which I think aligns with your view.
The notion that 'all we need is data and data is all there is' seems to summarize a lot of ML sentiments.
The notion that 'all we need is data and data is all there is' seems to summarize a lot of ML sentiments.
“I once built a face recognition system using 350 million images.”
Did this make any of you a little queasy?
Did this make any of you a little queasy?
Well noted! Explicitly: where does such database come from?
Frames of video could make the number sky-high like that without involving enormous numbers of people.
Right. Such as, extras in movies. Because, "350 million public faces" does not seem "cognitively digestible", but it could just be «350 million images» including large variations of the same faces, consistently with the "frames of video" idea.
data quality is important. every ai project i've worked on has started with visualizing the data and thinking about it.
it's easy to get complacent and focus on building big datasets. in practice, looking at the data often reveals issues sometimes in data quality and sometimes scope of what's in there (if you're missing key examples, it's simply not going to work).
most ml is actually data engineering.
it's easy to get complacent and focus on building big datasets. in practice, looking at the data often reveals issues sometimes in data quality and sometimes scope of what's in there (if you're missing key examples, it's simply not going to work).
most ml is actually data engineering.
Glad to see the term ML being used more often than AI in the comments as it looks like most "AI" models are trained for image classification.
Having said that, the idea of "doing more with less" sounds interesting and I wonder what it means exactly. Does it mean taking a dataset of 50 images and to create 1000s of synthetic images from it?
Yeah I was very interested about that point in particular. I think synthetic data is one of the ideas, but I got the sense that he also means helping to identify what makes a data set good, even if small. It looks like Andrew Ng is developing a platform for automatically detecting whether a dataset is suitable and, if not, what are the steps to improve it. A sort of automated ML consultant, allowing you to sell capabilities much cheaper than if you needed to consult an actual expert.
Pretty interesting. Mr. Ng claims that for some applications having a small set of quality data can be as good as using huge set of noisy data.
I wonder if, assuming the data is of highest quality, with minimal noise, having more data will matter for training or not. And if it matters, on what degree?
I wonder if, assuming the data is of highest quality, with minimal noise, having more data will matter for training or not. And if it matters, on what degree?
This is at the heart of the ML training problem.
In general you want to add more variants of data but not so much that the network doesn't get trained by them. Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) and prefer more of those.
Now, why not simply add everything? Well in general it takes too long to train.
In general you want to add more variants of data but not so much that the network doesn't get trained by them. Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) and prefer more of those.
Now, why not simply add everything? Well in general it takes too long to train.
> Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference)
How do you identify these images? It sounds like I'd need to build small models to see the variance but I'm hoping that there's a more scientific way?
How do you identify these images? It sounds like I'd need to build small models to see the variance but I'm hoping that there's a more scientific way?
It is relatively easy to turn small and accurate data to bigger and less accurate data with various forms of augmentation. The opposite is harder.
I can imagine that customizing AI solutions in an automated way is quite important, but writing that as the next wave is probably an overstatement.
Of course few shot learning is important for models, but for example for Pathways it was already part of the evaluation.
Of course few shot learning is important for models, but for example for Pathways it was already part of the evaluation.
[deleted]
For industrial application, there are already mature systems based on CV. For majority of those applications, there is no need for deep learning or multilayer CNN. Shocked to see Andrew Ng talking like a marketing guy.
What are some ML data annotation tools that guide you towards those data points where the model gets confused? I hear Prodigy does this. Any others?
What's the role of these tools? Can't a developer just write the code to get those data points?
At a first glance it seems like the hassle of integrating such a product into an existing ML codebase/pipeline is larger than solving the problem by hand.
At a first glance it seems like the hassle of integrating such a product into an existing ML codebase/pipeline is larger than solving the problem by hand.
What I mean is an annotation tool that interacts with the model itself in such a way that it will present to the user exactly those training examples next that will have the greatest impact in helping the model learn. So an annotation tool that provides a user interface for annotating data quickly (with keyboard shortcuts etc.). And looped into inference through the model to be trained, so you always get presented with the very training example that, out of the ones available, the model currently would be most unsure about.
Yeah that'd be great.
I also want cars that run on salt water.
I'm not saying that small data ai is equally impossible, but simply saying "we should make this better thing" isn't enough.
I also want cars that run on salt water.
I'm not saying that small data ai is equally impossible, but simply saying "we should make this better thing" isn't enough.
> simply saying "we should make this better thing" isn't enough.
Besides the references to his company which has customers and a product that already works on these principles the literature currently shows that this is very much possible if you dig into the correct niches. Besides the SOTA in few-shot and meta-learning it is possible to smartly choose the correct few samples for the network that yield the same results.
It has also been my primary focus for the past 5 years and the core of the company I founded.
Besides the references to his company which has customers and a product that already works on these principles the literature currently shows that this is very much possible if you dig into the correct niches. Besides the SOTA in few-shot and meta-learning it is possible to smartly choose the correct few samples for the network that yield the same results.
It has also been my primary focus for the past 5 years and the core of the company I founded.
> it is possible to smartly choose the correct few samples for the network that yield the same results.
And then, someone is using pretrained 500B model, and fine-tuning your few examples, and getting new SOTA.
And then, someone is using pretrained 500B model, and fine-tuning your few examples, and getting new SOTA.
They might get new SOTA because the metric is accuracy, but if the metric was accuracy weighted by sample efficiency, then SOTA would look a lot less impressive.
Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.
Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.
[deleted]
It's more of "this direction seems higher ROI than that direction", in particular quality vs quantity of data.
Already in 2018 SenseTime reported that for face recognition, clean dataset surpasses accuracy of 4x larger raw dataset.
https://arxiv.org/abs/1807.11649
Already in 2018 SenseTime reported that for face recognition, clean dataset surpasses accuracy of 4x larger raw dataset.
https://arxiv.org/abs/1807.11649
«Small data /ai/» is not "impossible", it is actually necessary: AI, opposed to this ML, implies perfectioned digestion of the input data.
Only, the article seemed to show a very conservative Ng about the algorithms, a focus on data management - so it's still ML.
Only, the article seemed to show a very conservative Ng about the algorithms, a focus on data management - so it's still ML.
I would say that Andrew Ng has some credibility in putting practice to his preaching.
Atleast someone's working on it.