Inside an AI 'brain' – What does machine learning look like?(graphcore.ai)
graphcore.ai
Inside an AI 'brain' – What does machine learning look like?
https://www.graphcore.ai/blog/what-does-machine-learning-look-like
28 comments
> The consensus ... is ...
> I'm not the only one within the research community to think this.
I should hope not if it's indeed the consensus :) Anyhow, I agree with your sentiment.
Comparisons with organic intelligence? Check.
Vague descriptions of new technologies? Check.
Flashy, uninformative graphics? Check.
Maybe I'm being too cynical.
> I'm not the only one within the research community to think this.
I should hope not if it's indeed the consensus :) Anyhow, I agree with your sentiment.
Comparisons with organic intelligence? Check.
Vague descriptions of new technologies? Check.
Flashy, uninformative graphics? Check.
Maybe I'm being too cynical.
"IEEE Spectrum: We read about Deep Learning in the news a lot these days. What’s your least favorite definition of the term that you see in these stories?
Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver."
[1] http://spectrum.ieee.org/automaton/robotics/artificial-intel...
Yann LeCun: My least favorite description is, “It works just like the brain.” I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain actually does. And describing it like the brain gives a bit of the aura of magic to it, which is dangerous. It leads to hype; people claim things that are not true. AI has gone through a number of AI winters because people claimed things they couldn’t deliver."
[1] http://spectrum.ieee.org/automaton/robotics/artificial-intel...
I wish this practical understanding of AI were understood by the public.
It seems that, when educating the public about AI in order to advance the field, we must first go through this phase where people do not understand and we risk taking a step back.
It seems that, when educating the public about AI in order to advance the field, we must first go through this phase where people do not understand and we risk taking a step back.
no you're exactly right
It looks like a brain and that's enough for me to jump on the hype train. I don't really care if it does nothing more.
well shit I'm sure I can take a RAM dump and soup up fancy graph code to make it look like a brain .. it still won't mean anything
Great, the do it, then we will A/B
I'm in love with data visualization / generative art.
I'm in love with data visualization / generative art.
One really contrived usecase may be "flame graphs" for high level diagnostics of learned models in a production pipeline, as you collect more data or modify features. But I don't imagine it'd be very good.
- the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure.
- requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score.
- if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.
- the images and false colors need to show some semblance of stability for a given network between epochs; and it needs to be robust against changing data or input structure.
- requiring visual inspection doesn't give you something you can automate with, unlike an evaluation score.
- if there is indeed a significant deviation in "MRI"-like scans between batches, its diagnostic utility ends there - it tells you nothing about what caused a change.
Aesthetically enjoyable and with an organic feel, the intricate detail suggests a lot of information is represented, yet Graphcore's diagram does not seem immediately apprehensible.
In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page).
On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer.
Alexnet's Layer 1 is Convolutional [a] :
or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/
In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity"
Does Chintala mean compute cores, Graphcore's IPUs ?
From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices."
What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ?
[edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers."
So it is a compiled computational graph just like Tensorflow's.
I cannot fathom it further and am unsure how or if to proceed.
Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry.
[a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides...
In the OP article's masthead, the clusters are labelled, this is AlexNet's computational graph from a Tensorflow description (depicted in full lower down the page).
On the right is "Conv1 11x11 forward [3 in, 64 out]" which suggests a Convolutional Layer with 3 inputs and 64 outputs. Forward, presumably, the direction of tensors flowing through the layer.
Alexnet's Layer 1 is Convolutional [a] :
• Images: 227x227x3
• F (receptive field size): 11
• S (stride) = 4
• Convlayer output: 55x55x96
or CS231n's page containing both structural and activation diagrams: http://cs231n.github.io/convolutional-networks/
In reply to your linked tweet [1], Chintala asserts the links are " compute/mem activity on their cores" and "connections between clusters are memory transfers/activity"
Does Chintala mean compute cores, Graphcore's IPUs ?
From Graphcore's page: "computational graphs are made up of vertices (think neurons) for the compute elements, connected by edges (think synapses), which describe the communication paths between vertices."
What do Graphcore's colours represent ? What is an IPU ? Is an IPU hardware like Google's TPU ?
[edit] Graphcore explains: "Our Poplar graph compiler has converted a description of the network into a computational graph of 18.7 million vertices and 115.8 million edges. This graph represents AlexNet as a highly-parallel execution plan for the IPU. The vertices of the graph represent computation processes and the edges represent communication between processes. The layers in the graph are labelled with the corresponding layers from the high level description of the network. The clearly visible clustering is the result of intensive communication between processes in each layer of the network, with lighter communication between layers."
So it is a compiled computational graph just like Tensorflow's.
I cannot fathom it further and am unsure how or if to proceed.
Counting the little dots and lines in cluster Conv1 and relating this to to 11x11 [3 in, 64 out] & Alexnet's 1st Layer could be a place to start an inquiry.
[a] http://vision.stanford.edu/teaching/cs231b_spring1415/slides...
I dont think thats important. What is important is that there might be insights. Who know what might come out of experimenting with it.
GP's point was that there are no insights here, just pretty pictures. We can see that they're complex, but that's about it. At least, that seems to be the agreement on this.
And my point is that there might be something coming out of trying to visualize this that we don't know of yet.
I wonder what is the motivation behind rendering the network inside a circle, and how to interpret from these charts... like: which neural ensemble is connected to which, etc.
Beautiful images for sure, but what do we learn from them?
Without any explanation of the questions you raise, this page is 99% marketing speak, and to me, next to useless.
Without any explanation of the questions you raise, this page is 99% marketing speak, and to me, next to useless.
Didn't want to sound as severe but yes... it's hard to decipher why they want it to make like a petri dish under a microscope.
I suppose the circle with its ideal properties (ie. smallest area to circumference ratio) allows for denser packing, or at least for easier calculation of packing because there are literally no corner cases
Historically, purpose-built processor architectures were always surpassed by general-purpose architectures. Think of Lisp Machines and x86, for a very relevant example of an architecture specifically designed for AI applications, that was quickly rendered obsolete by a general-purpose architecture.
Why graphcore is going to be any different is anybody's guess. Although, I admit the concept sounds cool on paper and the graph plots look pretty- I'd hang one on my wall for sure.
Why graphcore is going to be any different is anybody's guess. Although, I admit the concept sounds cool on paper and the graph plots look pretty- I'd hang one on my wall for sure.
We're in a different historical moment, though. GPUs are specialized hardware for linear algebra, and have kind of taken over. TPUs are similarly specialized. The big difference is that Moore's law seems to be stalled out for standard, general purpose processing; as a result there's a lot to be gained in specializing.
Now whether these particular professors are worth anything is a different question...
Now whether these particular professors are worth anything is a different question...
*processors
General purpose always won because a) the demands wasn't great enough b) the steamroller of moore's law helped general purpose better and c) previous special purpose chips were maybe 10x better, not 100-250X.
Can somebody explain this to me?
Unlike a scalar CPU or a vector GPU, the Graphcore Intelligent Processing Unit (IPU) is a graph processor.
Is this just marketing mumbo-jumbo? I don't understand how a "graph processor" would look any different than a vector processor.
Is this just marketing mumbo-jumbo? I don't understand how a "graph processor" would look any different than a vector processor.
Founder of a different deep learning chip startup here. They're talking about deep learning's computational graphs. The only differences between that and a general purpose program graph is that a) the (general) absence of control flow and b)the operations are all arithmetic and generally predictably so.
So basically yeah, it's marketing.
So basically yeah, it's marketing.
a gpu is optimized for 2 and 3d, graphs might represent higher topologies?
Here is some explanation:
https://www.graphcore.ai/blog/graph-computing-for-machine-in...
Seems like PR bait. Tim Davis' page is much much nicer.
http://www.cise.ufl.edu/research/sparse/matrices/synopsis/
I think Trefethen too has nice visualization like this (or maybe that was the spectral thing).
http://www.cise.ufl.edu/research/sparse/matrices/synopsis/
I think Trefethen too has nice visualization like this (or maybe that was the spectral thing).
Wow that looks organic, it looks like a bacteria learning and adapting to its surroundings.
I'm not the only one[1] within the research community to think this.
If you find any insights from this, I'd honestly first be surprised and then second be interested to know what insights you gleamed from it.
Background: researcher who publishes papers in deep learning.
[1]: https://twitter.com/jackclarkSF/status/834461913262157824 (thread containing a member of OpenAI who specializes in communicating complex machine learning topics to the media and a primary developer of PyTorch / member of Facebook's AI Research lab)