When to assume neural networks can solve a problem(blog.cerebralab.com)
blog.cerebralab.com
When to assume neural networks can solve a problem
https://blog.cerebralab.com/When_to_assume_neural_networks_can_solve_a_problem
43 comments
At least on my account, it seems like Amazon has given up on recommendations and is just suggesting I buy the same things again.
I wouldn't be shocked if it's actually quite a bit more accurate then real predictions, but it does lead to funny things like "You already bought this book? How about a second copy?"
I wouldn't be shocked if it's actually quite a bit more accurate then real predictions, but it does lead to funny things like "You already bought this book? How about a second copy?"
I wonder if the same book recommendations are happening because people might just be buying products like these for gifts. Something like I liked this book, so you should read it to.
This reminds me of an oldie, but goodie: http://thedailywtf.com/articles/No%2C_We_Need_a_Neural_Netwo...
Wow, I had never read this gem before. Made me laugh, thanks for sharing this piece! I'm surprised they did not ask him to log everything in a blockchain :)
No Quack.
Here is a take from Google AI on when to use ML:
https://developers.google.com/machine-learning/problem-frami...
"You should understand the problem well enough to list some potential solutions to test first to find the best model" and "it shouldn't be used for a problem that conventional programming can solve" like hard coding a symbolic rule.
"You should understand the problem well enough to list some potential solutions to test first to find the best model" and "it shouldn't be used for a problem that conventional programming can solve" like hard coding a symbolic rule.
Unless it's fizzbuzz:
https://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/
https://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/
it seems that neural networks have problem for computing the maximum function, and a human can compute the maximum easily, so it seems that the three heuristic rules don't work in this case.
(1) https://datascience.stackexchange.com/questions/56676/can-ma...
(1) https://datascience.stackexchange.com/questions/56676/can-ma...
You're referring to about whether a generic one-size-fits-all model will do well, but ML is full of bespoke models. It would be simple to build a neural network that can compute (and differentiate through) the max function to within some arbitrary epsilon, even though the most generic model (feed forward network) won't do great.
> l (feed forward network) won't do great.
See my answer below, in the case of this problem a generic feed-forward network, even a simple one, will work.
Not any ffn, but assuming you are using an efficient architecture search it will probably find one that works.
There's other numerical problems where this doesn't hold but that's another story.
See my answer below, in the case of this problem a generic feed-forward network, even a simple one, will work.
Not any ffn, but assuming you are using an efficient architecture search it will probably find one that works.
There's other numerical problems where this doesn't hold but that's another story.
You demonstrated it for a reeeeeeeallly constrained version of the problem. Do you expect your solution would generalize to many lists? Because it would be easy to make a neural network that does, while your toy example (and larger generalizations) probably won't generalize super well.
x_i = ith list element from list x
y = sum(x_i * softmax(k * x)_i)
This one parameter, arbitrarily wide network one will get arbitrarily close to the max function.
This is a super toy version of why attention is so effective. It can pick stuff.
x_i = ith list element from list x
y = sum(x_i * softmax(k * x)_i)
This one parameter, arbitrarily wide network one will get arbitrarily close to the max function.
This is a super toy version of why attention is so effective. It can pick stuff.
That is untrue,
Here's a code example (actually took me ~20 minutes to get it "right" so I'll admit it's not the most trivial problem)... it includes seeds so that you can replicate locally (it should hit 100% accuracy all the time on the 1200 examples testing set reliably by about epoch 700):
'''
import torch import random from sklearn.metrics import accuracy_score
random.seed(61) torch.manual_seed(61)
X = [[random.random() for x in range(2)] for x in range(2000)] X_train = torch.FloatTensor(X[0:800]).cuda() X_test = torch.FloatTensor(X[800:]).cuda() X = torch.FloatTensor(X)
Y = [] for x in X: y = [0] * len(x) y[torch.argmax(x)] = 1 Y.append(y)
Y_train = torch.FloatTensor(Y[0:800]).cuda() Y_test = torch.FloatTensor(Y[800:]).cuda()
shape = [2,2] layers = [] for ind in range(len(shape) - 1): layers.append(torch.nn.Linear(shape[ind],shape[ind+1],bias=False))
net = torch.nn.Sequential(layers).cuda()
optim = torch.optim.SGD(net.parameters(), lr=1) criterion = torch.torch.nn.CrossEntropyLoss()
dataset = torch.utils.data.TensorDataset(X_train, Y_train) dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=10)
for i in range(pow(10,6)): for X,Y in dataloader: Yp = net(X) loss = criterion(Yp, Y.max(1).indices) loss.backward() optim.step() optim.zero_grad()
This is as basic as you can get, predict the max out of 2 numbers, only uses a total of 4 node:
2 inputs (the 2 number) -> 2 outputs (the index of the maximum numbers). just 2 weight being optimize, no biases no nothing, as simple an implementation as you can get in terms of size.
There's also way to do it (apparently) where instead of treating it as "find the max index" you treat it as "output the maximum number": https://www.quora.com/Can-deep-neural-networks-learn-the-min...
But the approach I have will generalize to e.g. "Find the max of 5 or 100 or 1000 numbers" (although I assume it might take some time)
And overall you have no guarantee, that's why I qualified the statement and didn't say "Literally any imaginable problem that a human can solve without context".
To some extent it also matter how you encoder your number, you can train a 10000000000 parameter FCNN with RELU activations until the end of time to learn a simple mutliplication, and it won't be able to do so if you don't log encode your numbers or use some encoding or activation that means `` can be transposed in the `+` operations being done inside the nodes to combine the outputs... because that's outside of the scope of mathematics that given netwrok can do.
But, unless you are specifically trying to come up with an edge case and are instead looking at real world problems and trying to design the network in such a way as to best handle them (and this doesn't have to be all manual, you can use various NAS techniques), the rule will hold most of the time I believe.
Here's a code example (actually took me ~20 minutes to get it "right" so I'll admit it's not the most trivial problem)... it includes seeds so that you can replicate locally (it should hit 100% accuracy all the time on the 1200 examples testing set reliably by about epoch 700):
'''
import torch import random from sklearn.metrics import accuracy_score
random.seed(61) torch.manual_seed(61)
X = [[random.random() for x in range(2)] for x in range(2000)] X_train = torch.FloatTensor(X[0:800]).cuda() X_test = torch.FloatTensor(X[800:]).cuda() X = torch.FloatTensor(X)
Y = [] for x in X: y = [0] * len(x) y[torch.argmax(x)] = 1 Y.append(y)
Y_train = torch.FloatTensor(Y[0:800]).cuda() Y_test = torch.FloatTensor(Y[800:]).cuda()
shape = [2,2] layers = [] for ind in range(len(shape) - 1): layers.append(torch.nn.Linear(shape[ind],shape[ind+1],bias=False))
net = torch.nn.Sequential(layers).cuda()
optim = torch.optim.SGD(net.parameters(), lr=1) criterion = torch.torch.nn.CrossEntropyLoss()
dataset = torch.utils.data.TensorDataset(X_train, Y_train) dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=10)
for i in range(pow(10,6)): for X,Y in dataloader: Yp = net(X) loss = criterion(Yp, Y.max(1).indices) loss.backward() optim.step() optim.zero_grad()
if i > 500:
optim = torch.optim.SGD(net.parameters(), lr=0.002)
if i % 20 == 0:
Yp = net(X_test)
print('Training loss: ', loss.item())
print(f'\nAccuracy score for epoch {i}:')
print(accuracy_score(Yp.max(1).indices.tolist(),Y_test.max(1).indices.tolist()))
This is as basic as you can get, predict the max out of 2 numbers, only uses a total of 4 node:
2 inputs (the 2 number) -> 2 outputs (the index of the maximum numbers). just 2 weight being optimize, no biases no nothing, as simple an implementation as you can get in terms of size.
There's also way to do it (apparently) where instead of treating it as "find the max index" you treat it as "output the maximum number": https://www.quora.com/Can-deep-neural-networks-learn-the-min...
But the approach I have will generalize to e.g. "Find the max of 5 or 100 or 1000 numbers" (although I assume it might take some time)
And overall you have no guarantee, that's why I qualified the statement and didn't say "Literally any imaginable problem that a human can solve without context".
To some extent it also matter how you encoder your number, you can train a 10000000000 parameter FCNN with RELU activations until the end of time to learn a simple mutliplication, and it won't be able to do so if you don't log encode your numbers or use some encoding or activation that means `` can be transposed in the `+` operations being done inside the nodes to combine the outputs... because that's outside of the scope of mathematics that given netwrok can do.
But, unless you are specifically trying to come up with an edge case and are instead looking at real world problems and trying to design the network in such a way as to best handle them (and this doesn't have to be all manual, you can use various NAS techniques), the rule will hold most of the time I believe.
You are right. Also the Universal Approximation theorem (1) for neural networks guarantees that neural networks can approximate continuous function on compact subsets of R^n, in this case max(x,y).
argmax([x,y]) = (sign(x[0]-x[1])+1)/2
Going beyond continuous functions, can deep learning be used for primality test?
(1) https://en.wikipedia.org/wiki/Universal_approximation_theore...
argmax([x,y]) = (sign(x[0]-x[1])+1)/2
Going beyond continuous functions, can deep learning be used for primality test?
(1) https://en.wikipedia.org/wiki/Universal_approximation_theore...
https://escholarship.org/content/qt5sg7n4ww/qt5sg7n4ww.pdf
> A long-standing difficulty for connectionism has been to implement compositionality, the idea of building a knowledge representation out of components such that the meaning arises from the meanings of the individual components and how they are combined. Here we show how a neural-learning algorithm, knowledge-based cascade-correlation (KBCC), creates a compositional representation of the prime-number concept and uses this representation to decide whether its input n is a prime number or not. KBCC conformed to a basic prime-number testing algorithm by recruiting source networks representing division by prime numbers in order from smallest to largest prime divisor up to √n. KBCC learned how to test prime numbers faster and generalized better to untrained numbers than did similar knowledge-free neural learners. The results demonstrate that neural networks can learn to perform in a compositional manner and underscore the importance of basing learning on existing knowledge.
But again, I think things such as prime number tests are the exact kind of edge cases where one needs too many heuristics built into the model for it to be practical to use.
But I think something like a prime test is not included under the definition I gave anyway, because the idea of "prime" actually implies a lot of context.
You can take a baby and he will be able to classify images, you can take a human that speaks a language with no concept of numbers and he will be able to play or sing music and distinguish patterns in it.
You can't talk about "prime" without a mathematical apparatus that takes years for humans to understand. However, since we learn it as such an early age, it ends up in the background.
Granted, that could be said about almost any cognitive ability (the fact that there's a lot of "subconscious context" required to use it).... so I don't know.
> A long-standing difficulty for connectionism has been to implement compositionality, the idea of building a knowledge representation out of components such that the meaning arises from the meanings of the individual components and how they are combined. Here we show how a neural-learning algorithm, knowledge-based cascade-correlation (KBCC), creates a compositional representation of the prime-number concept and uses this representation to decide whether its input n is a prime number or not. KBCC conformed to a basic prime-number testing algorithm by recruiting source networks representing division by prime numbers in order from smallest to largest prime divisor up to √n. KBCC learned how to test prime numbers faster and generalized better to untrained numbers than did similar knowledge-free neural learners. The results demonstrate that neural networks can learn to perform in a compositional manner and underscore the importance of basing learning on existing knowledge.
But again, I think things such as prime number tests are the exact kind of edge cases where one needs too many heuristics built into the model for it to be practical to use.
But I think something like a prime test is not included under the definition I gave anyway, because the idea of "prime" actually implies a lot of context.
You can take a baby and he will be able to classify images, you can take a human that speaks a language with no concept of numbers and he will be able to play or sing music and distinguish patterns in it.
You can't talk about "prime" without a mathematical apparatus that takes years for humans to understand. However, since we learn it as such an early age, it ends up in the background.
Granted, that could be said about almost any cognitive ability (the fact that there's a lot of "subconscious context" required to use it).... so I don't know.
[deleted]
> 3. A neural network can solve problems that a human can solve with small-sized datapoints and little to no context
I like this rule, because it is actionable. Its easy to take a sample of your data, (in the case of images) lower the quality to 20x20 pixels and get see if you, or another person can do the classification task. Audio could be handled similarly.
I like this rule, because it is actionable. Its easy to take a sample of your data, (in the case of images) lower the quality to 20x20 pixels and get see if you, or another person can do the classification task. Audio could be handled similarly.
When a neural network solved a problem, it means it found the algebraic function it needs to solve the problem.
> Once it found the algebraic function, there is no need to run the problem to the neural network. We can sideload the problem to a simple program that takes input and gives an output. This strategy can help to free up the GPU for next set of problem.
> Once it found the algebraic function, there is no need to run the problem to the neural network. We can sideload the problem to a simple program that takes input and gives an output. This strategy can help to free up the GPU for next set of problem.
How would one extract an algebraic function from the network’s weights? Why would the function the network approximates necessarily be algebraic?
The function would simply be a chain of multiplications and (typically) non-linear transformations.
For classification problems, there is often a final non-linear transformation, like a softmax.
Example with ReLu activation:
Output_i = max((input_i * weight_i),0)
For classification problems, there is often a final non-linear transformation, like a softmax.
Example with ReLu activation:
Output_i = max((input_i * weight_i),0)
I don't understand. The example you gave is not an algebraic function.
You are correct. It is possible to construct a linear neural network, where an algebraic function could be extracted, but in practice, almost all networks use non-linear activation functions.
In the case of a linear network, the function would be a dot product between the input and the weights: 𝑥1𝑤1+𝑥2𝑤2+𝑥3𝑤3 ... for all inputs (xi) and weights (wi)
In the case of a linear network, the function would be a dot product between the input and the weights: 𝑥1𝑤1+𝑥2𝑤2+𝑥3𝑤3 ... for all inputs (xi) and weights (wi)
I don't understand why the author thinks that the PAC learning framework is not useful. PAC gives quick back-of-napkin math for estimating how long it might take to train a network, what kind of quality level is acceptable for inputs, etc.
I disagree, PAC bounds are usually way too loose to be useful in practice when it comes to NNs. I think part of the reason is that the way we measure model complexity for NNs does not correspond very well with their ability to generalize, as we've seen with recent papers like deep double descent [1].
Anecdotally, I was of the main authors of the AdaNet framework [2] where we used (approximate) Rademacher complexity (closely related to PAC) to bound the complexity of our learned ensemble models. We never got good results using the complexity measure and would basically turn it off for any practical problem we solved.
[1]: https://openai.com/blog/deep-double-descent
[2]: https://github.com/tensorflow/adanet
Anecdotally, I was of the main authors of the AdaNet framework [2] where we used (approximate) Rademacher complexity (closely related to PAC) to bound the complexity of our learned ensemble models. We never got good results using the complexity measure and would basically turn it off for any practical problem we solved.
[1]: https://openai.com/blog/deep-double-descent
[2]: https://github.com/tensorflow/adanet
Correct me if I'm wrong, but doesn't PAC on real data usually just tells you that "it doesn't know the answer"? Like, it says the equivalent to: "I think this is the answer, but it might be miles off and I give it 50% it is even more than miles off." I found that it is able to give hard guarantees, by being very loose on the bounds. Do you have a real-world example where it actually gave something useful? I'm genuine curious, because I discarded it early on, but am willing to revisit it.
PAC does not provide these types of estimates in most practical situations.
What is the current state of NN in the context of AI state of the art?
I've been hearing about deep learning revolutionizing "everything" for the past 8 years. So, can someone name me any significant impact it made on figuring out what to do about the recent epidemic? If not, I think it's worth reflecting on what value we get out of the technology that sucked up so much of our intellectual, financial and computational resources.
If you're listening to people who claim any one technology solves "everything" --- and believing them --- then you are the fool.
There's no silver bullet. Deep Learning has made huge strides in noise reduction, image and video processing (self driving cars), lidar processing, medical imaging, etc, but it's not made any progress in knitting, tooth brushing, dog walking, or many other tasks ill suited for deep learning - so clearly not everything.
There's no silver bullet. Deep Learning has made huge strides in noise reduction, image and video processing (self driving cars), lidar processing, medical imaging, etc, but it's not made any progress in knitting, tooth brushing, dog walking, or many other tasks ill suited for deep learning - so clearly not everything.
Your general point is valid, but I can't help pointing out that knitting may not belong in that second list as much as the others.
Computational textile work is a niche, but fascinating, area of research. Knitting and crocheting are highly technical crafts, and there are opportunities to use modern computational approaches, including deep learning, in them.
Deep Knitting: https://arxiv.org/pdf/1902.02752.pdf
I'm not going to read this paper in detail, it may not be a great example of a successful deep learning application - but at least it's not an absurd idea.
Maybe not surprising if you're aware of the place of the Jacquard loom in the history of computer science.
Computational textile work is a niche, but fascinating, area of research. Knitting and crocheting are highly technical crafts, and there are opportunities to use modern computational approaches, including deep learning, in them.
Deep Knitting: https://arxiv.org/pdf/1902.02752.pdf
I'm not going to read this paper in detail, it may not be a great example of a successful deep learning application - but at least it's not an absurd idea.
Maybe not surprising if you're aware of the place of the Jacquard loom in the history of computer science.
That's been the general vibe the popular tech press has been spouting for awhile. That NNs are somehow a major step towards AGI, which'll 'capture the lightcone of the future' to quote Sam Altman.
At some point, we'll realize that AGI is the modern alchemy. Ironically, alchemy's penultimate goal was to construct a homoculus, which is essentially the goal of AGI.
https://en.wikipedia.org/wiki/Homunculus#Alchemy
"The allegorical text suggests to the reader that the ultimate goal of alchemy is not chrysopoeia, but it is instead the artificial generation of humans."
At some point, we'll realize that AGI is the modern alchemy. Ironically, alchemy's penultimate goal was to construct a homoculus, which is essentially the goal of AGI.
https://en.wikipedia.org/wiki/Homunculus#Alchemy
"The allegorical text suggests to the reader that the ultimate goal of alchemy is not chrysopoeia, but it is instead the artificial generation of humans."
> That's been the general vibe the popular tech press has been spouting for awhile
This is such a vague generalisation, I cant do anything with it, except maybe how about "nullius in verba", on the word of noone, or "think for yourself"
This is such a vague generalisation, I cant do anything with it, except maybe how about "nullius in verba", on the word of noone, or "think for yourself"
It's not revolutionizing everything in the field of AI. It's "only" good at prediction in domains where data is poorly structured, such as images, audio, video, text and some games (which is a lot to be fair, hence the publicity it got in the recent years). It's not that great on structured data or on small datasets, and is not really adapted to anything not related to raw, "black-boxed" prediction.
I'm not sure what you'd expect that tool to do regarding the current epidemic.
I'm not sure what you'd expect that tool to do regarding the current epidemic.
Not good on "small datasets", well in theory, but in practice transfer learning given a similar training distribution can take you quite far.
I've been hearing about modern JavaScript development revolutionizing "everything" for the past 8 years. So, can someone name me any significant impact it made on figuring out what to do about the recent epidemic? If not, I think it's worth reflecting on what value we get out of the technology that sucked up so much of our intellectual, financial and computational resources.
This is a nonsensical response on two levels.
First, you're missing the obvious fact that the response to this pandemic is largely dictated by data processing. People are modeling the spread of the virus, trying to make predictions about the optimal policy, trying to figure out infection rates without full testing and evaluating promising drugs. All of those things relate to modeling and data analysis. Deep learning was touted to revolutionize exactly those two things.
Second, web technologies are making an obvious impact by allowing (some, many) people to stay in touch, stay informed, keep working and buy stuff without exposing themselves to crowded places. This constitutes visible and significant positive impact.
First, you're missing the obvious fact that the response to this pandemic is largely dictated by data processing. People are modeling the spread of the virus, trying to make predictions about the optimal policy, trying to figure out infection rates without full testing and evaluating promising drugs. All of those things relate to modeling and data analysis. Deep learning was touted to revolutionize exactly those two things.
Second, web technologies are making an obvious impact by allowing (some, many) people to stay in touch, stay informed, keep working and buy stuff without exposing themselves to crowded places. This constitutes visible and significant positive impact.
Well, I haven't had a remotely relevant Amazon recommendation in 5 to 10 years. Unless you count read a book by the exact same author as a useful recommendation.