Finding Language in the Brain(thereader.mitpress.mit.edu)
thereader.mitpress.mit.edu
Finding Language in the Brain
https://thereader.mitpress.mit.edu/finding-language-in-the-brain/
19 comments
There was such a theory, developed in the 1960s and 1970s, I think, although I can remember neither the name of the linguist (I don't think it caught on broadly, so it really was just one researcher) nor the name of the theory.
I think it foundered for a number of reasons, one of which was that it had--so far as I know--no account of the creativity of humans "doing" language. That is, we can understand and speak any number of sentences we've never heard before, and it's hard (or impossible) to imagine that all those novel sentences were ever in the brain, or that new nerve connections grew so rapidly.
For example, I can tell you that I've invented a fugelizer, and that it is useful for turning bryles into fugels. Now I've just used three words that you've never heard before, but you can sort of make sense of what I've said. But clearly there's no 'fugelizer' neuron in your brain, etc.
I think it foundered for a number of reasons, one of which was that it had--so far as I know--no account of the creativity of humans "doing" language. That is, we can understand and speak any number of sentences we've never heard before, and it's hard (or impossible) to imagine that all those novel sentences were ever in the brain, or that new nerve connections grew so rapidly.
For example, I can tell you that I've invented a fugelizer, and that it is useful for turning bryles into fugels. Now I've just used three words that you've never heard before, but you can sort of make sense of what I've said. But clearly there's no 'fugelizer' neuron in your brain, etc.
Well I have no idea what a fugelizer or a fugel or a bryles are, but those are just as good abstract "things" as any other and so I can just as easily fit them into the pattern of (thing) and (other thing) --(pro existence of)--> (third thing). I can also safely assume that the reason for things existing eventually circuits down to "someone wants it to be this way". So, why would anyone rather their bryles be fugels?
(I can figure out other semantics of how bryles and fugels relate to other things I know about later. I don't need a full map of concept relations before I start connecting up the circuit and seeing what it does).
(I can figure out other semantics of how bryles and fugels relate to other things I know about later. I don't need a full map of concept relations before I start connecting up the circuit and seeing what it does).
Not too far off base from my understanding. But the first few layers are like firmware they provide the neural interface between hardware (reflexes, processing sensations, etc) and the rest of thinking. So it’s a system that it’s very deeply tied to how we perceive and interact with our bodies and through them the world.
And in fact the first level or two of the hierarchy happen outside the brain in the nervous system or spine (unless hearing or sight related obvs)
There’s one piece missing though and that’s goals/control variable. My thoughts about this is how we encode things and act of things. When we recognize a tree there’s a set of nodes that activate then some higher layer recognizes trust as a tree and a higher layer still decides how to use that information or how to interact with those nodes (it sets a desired state. Ie. Walk to the tree, write the word, recognize the picture, etc).
Take a look into Hierarchical Perceptual Control Theory.
And in fact the first level or two of the hierarchy happen outside the brain in the nervous system or spine (unless hearing or sight related obvs)
There’s one piece missing though and that’s goals/control variable. My thoughts about this is how we encode things and act of things. When we recognize a tree there’s a set of nodes that activate then some higher layer recognizes trust as a tree and a higher layer still decides how to use that information or how to interact with those nodes (it sets a desired state. Ie. Walk to the tree, write the word, recognize the picture, etc).
Take a look into Hierarchical Perceptual Control Theory.
One interesting implication of this is that language is not something separate in the brain.
It is a higher level abstraction over the existing processes and reuses all of the existing machinery. So our conception of the word tree would translate to other animals brains directly, they just lack the higher level abstractions of language, consciousness, and understanding that others think and communicate.
So the additional nodes collected under the “tree” node (or nodes) would be cleared and repurposed because it’s useless information to the other animal.
It is a higher level abstraction over the existing processes and reuses all of the existing machinery. So our conception of the word tree would translate to other animals brains directly, they just lack the higher level abstractions of language, consciousness, and understanding that others think and communicate.
So the additional nodes collected under the “tree” node (or nodes) would be cleared and repurposed because it’s useless information to the other animal.
How does the parse tree get represented? Who or what chooses the mapping? They have to be different every sentence. E.g., the word "can" maps onto many different meanings and functions. Your question is about internal representation, but skips the process, and is about a representation that isn't the end of the process either.
The idea, common to many of competing theories of human NLP, is that language is to a large extent modular (e.g. recognizing words can be done independent of e.g. a sentence context), while on the other hand, the context (often thought of as feedback from other modules/brain parts) has some influence over it (e.g. the recognition of a misspelled word is easily influenced by the sentence it appears in). The "layers" of language processing then are word, structure, meaning, and pragmatic. The brain being incredibly interconnected, this picture is simplistic, but both behavioral experiments and neurological findings support it to a large extent: it's not one big neural soup. There are other, related processes, which are often left out when talking about language, but e.g. gesture and visual information definitely play a role as well.
The parse tree does exist somewhere in the brain, but it cannot map to connections. Instead, it will have to encode some relation between working memory, the large (and largely ununderstood) space of our mind that contains the concepts, and the role in the sentence and further context. Encoding the "pointer address" (because that's what your model sounds like) of the word is simply not enough. It will be passed along at some point, but is in itself insufficient. E.g. in I want the red ball, not the blue ball, the receiver needs to distinguish objects referenced by the exact same word with the exact same meaning. Having a pointer to the lexical concept ball will not help.
I think showing "that its wrong" only gives insights into your own mental model, not into language processing in general. It is too undeveloped for further discussion.
The idea, common to many of competing theories of human NLP, is that language is to a large extent modular (e.g. recognizing words can be done independent of e.g. a sentence context), while on the other hand, the context (often thought of as feedback from other modules/brain parts) has some influence over it (e.g. the recognition of a misspelled word is easily influenced by the sentence it appears in). The "layers" of language processing then are word, structure, meaning, and pragmatic. The brain being incredibly interconnected, this picture is simplistic, but both behavioral experiments and neurological findings support it to a large extent: it's not one big neural soup. There are other, related processes, which are often left out when talking about language, but e.g. gesture and visual information definitely play a role as well.
The parse tree does exist somewhere in the brain, but it cannot map to connections. Instead, it will have to encode some relation between working memory, the large (and largely ununderstood) space of our mind that contains the concepts, and the role in the sentence and further context. Encoding the "pointer address" (because that's what your model sounds like) of the word is simply not enough. It will be passed along at some point, but is in itself insufficient. E.g. in I want the red ball, not the blue ball, the receiver needs to distinguish objects referenced by the exact same word with the exact same meaning. Having a pointer to the lexical concept ball will not help.
I think showing "that its wrong" only gives insights into your own mental model, not into language processing in general. It is too undeveloped for further discussion.
>The parse tree does exist somewhere in the brain
Is that even clear? Parse trees are a discretization of stochastic grammars, for example, with probabilities of "production rules" set to either 1 (so you get the right hand side rewrite) or 0 (so you don't get that as an option).
Disclaimer: worked on stochastic grammars for computation in recognizers 20 years ago, not sure how the defs might have drifted since then.
Is that even clear? Parse trees are a discretization of stochastic grammars, for example, with probabilities of "production rules" set to either 1 (so you get the right hand side rewrite) or 0 (so you don't get that as an option).
Disclaimer: worked on stochastic grammars for computation in recognizers 20 years ago, not sure how the defs might have drifted since then.
Nope. The representation in the brain seems to be like a parse tree, but it is a statistical AST, not how we think of it with computers.
If you have a basis in stochastic grammar, you should look into Operator Grammar. It is a very compelling description of grammar based on set theory and maps very neatly to the best neuroscience I'm aware of.
If you have a basis in stochastic grammar, you should look into Operator Grammar. It is a very compelling description of grammar based on set theory and maps very neatly to the best neuroscience I'm aware of.
oh cool...thanks for the recommendation, as i'm stale in that space for sure
This is an extremely dismissive comment, plus you are way off basis that it is underdeveloped. It is just new/underdeveloped for OP.
Literal books have been written about this exact idea by Zellig Harris. The reason it is not more well known is that Chomsky (his student) became famous and reinterpreted a couple of his ideas (using mathematical logic for the basis of his work rather than Harris's set-theoretic base). But while Chomsky had to keep creating castles in the sky (including the idea of universal grammar/external metalanguage) to patch the holes in his theory Harris's theory needs none of that and is self-organizing and self-sufficient.
An alternate (and consistent with OPs idea) list of layers of processing a language.
1. Phonemic distinctions.
2. Words and morphemes, with their main meanings. (The word with its morphemes is an umbrella that also encompasses gesture, visual, and any other perceptual information related to the word. The words we use are not separate entities from our experiences. In the same way, mirror neurons aren't a thing. We mirror because we only have one set of machinery, and so to understand the other person is literally to hallucinate experiencing it to some degree.)
3. Word dependencies (the argument requirement of each word).
4. The selection of each word (the dependencies that have greater than average likelihood).
5. The canonical or preferred linearization (word order) and its alternatives.
6. The main reductions (variant shapes of words), their domain (a particular word or all words in a position) and the conditions for applying them.
Linearization is one of the last steps of producing a sentence. According to Operator grammar theory, most language complications come from the simplifications we produce as shortcuts (such as pronouns, contractions, conjunctions, etc).
So your first two questions are just the memory of the words/ideas/objects/etc combined with an operator and our perceptions in context do the choosing. Very much like an AST, but statistical (for word likeliness), as opposed to deterministic.
The most differences start to arise when we start the process of moving that sentence externally. We apply reductions and linearization which are language specific.
The intersection between HPCT (Hierarchical Perceptual Control Theory) and Operator Grammar explains the details pretty well. And Zellig Harris presents a complete analysis of English Grammar in "A Grammar of English on Mathematical Principles". It is a very different way of looking at language grammar, which makes much more sense to me biologically he spends significant time showing it can also reproduce traditional grammar. I.e. Traditional grammar could be viewed as a change of basis from Operator grammar.
Take a look at An embodied grammar of words[1] for an overview. [1] https://www.researchgate.net/publication/324362482_An_embodi...
Literal books have been written about this exact idea by Zellig Harris. The reason it is not more well known is that Chomsky (his student) became famous and reinterpreted a couple of his ideas (using mathematical logic for the basis of his work rather than Harris's set-theoretic base). But while Chomsky had to keep creating castles in the sky (including the idea of universal grammar/external metalanguage) to patch the holes in his theory Harris's theory needs none of that and is self-organizing and self-sufficient.
An alternate (and consistent with OPs idea) list of layers of processing a language.
1. Phonemic distinctions.
2. Words and morphemes, with their main meanings. (The word with its morphemes is an umbrella that also encompasses gesture, visual, and any other perceptual information related to the word. The words we use are not separate entities from our experiences. In the same way, mirror neurons aren't a thing. We mirror because we only have one set of machinery, and so to understand the other person is literally to hallucinate experiencing it to some degree.)
3. Word dependencies (the argument requirement of each word).
4. The selection of each word (the dependencies that have greater than average likelihood).
5. The canonical or preferred linearization (word order) and its alternatives.
6. The main reductions (variant shapes of words), their domain (a particular word or all words in a position) and the conditions for applying them.
Linearization is one of the last steps of producing a sentence. According to Operator grammar theory, most language complications come from the simplifications we produce as shortcuts (such as pronouns, contractions, conjunctions, etc).
So your first two questions are just the memory of the words/ideas/objects/etc combined with an operator and our perceptions in context do the choosing. Very much like an AST, but statistical (for word likeliness), as opposed to deterministic.
The most differences start to arise when we start the process of moving that sentence externally. We apply reductions and linearization which are language specific.
The intersection between HPCT (Hierarchical Perceptual Control Theory) and Operator Grammar explains the details pretty well. And Zellig Harris presents a complete analysis of English Grammar in "A Grammar of English on Mathematical Principles". It is a very different way of looking at language grammar, which makes much more sense to me biologically he spends significant time showing it can also reproduce traditional grammar. I.e. Traditional grammar could be viewed as a change of basis from Operator grammar.
Take a look at An embodied grammar of words[1] for an overview. [1] https://www.researchgate.net/publication/324362482_An_embodi...
Great post. I am far too much of a linguistic neophyte to have any input but in trying to get a grasp on the field it does strike me that Chomsky will be a real time example of science advancing one funeral at a time.
I wouldn't be shocked if he becomes the archetype of this process for future generations. Hopefully we can find a word someday for this process of deification of an individual that holds back an entire field while the deity is alive.
I wouldn't be shocked if he becomes the archetype of this process for future generations. Hopefully we can find a word someday for this process of deification of an individual that holds back an entire field while the deity is alive.
>How does the parse tree get represented?
>Your question is about internal representation, but skips the process, and is about a representation that isn't the end of the process either.
I think you missed the essence of my conjecture. I'm not "skipping the process". I'm speculating that there is surprisingly little process to be done at all because language is "close to the metal" (to borrow an analogy).
Here's a simple but limited language. We can have electronic components like batteries and resistors and inductors as our basic noun objects, and our relational verb words are all just telling us if subjects and objects are connected in parallel or series. The only adjectives are numerical pre-fixes for amounts of voltage/resistance/whatever. A "." will suffice as a grammatical particle to mark "end of object". Its a bit verbose and cumbersome, but I can describe whatever circuit I want in these terms. In fact, the parse tree of this grammar can literally encode any loop-free circuit, with sibling branches in the parse tree corresponding to parallel connections and nested branches in the tree corresponding to series connections. I could literally use this grammar and vocab as a file format for an FPGA if I wanted to. One doesn't even need a fully general computation or additional reasoning abilities to "understand" this circuit language. A machine that physically builds the connections according to the grammatical rules as the words stream in would suffice. The semantics of this toy language don't require any further processing or translation.
Now here's the toy brain I envision. Purpose follows a tree structure. Things happen and things exist because someone wills them to be. When you get paid to work, the work serves the will of the company. When you do it for yourself, its your free will. When your computer does something, its because of your will (never its own free will). Each person is both a source of will power and a thing that exists as a matter of ground truth. Thus we can think of a person as a battery in a simplified analog model of our world.
For example you can describe a short circuit of one of these person-batteries with a mutually recursive circuit-statement: "I am because I want to be. I want to be because I am." This is a vacuous truth. You can also make more sophisticated paths from will power (want) to common ground (facts) as a way to encode a more complex scenario. All you have to do is put intermediate things in the path between "I want" and "I am". "I want [A] because [B] so that [C] or [D] which leads to [E] ... so that I continue to be." The full semantics of the objects in the broader context may or may not be known, but it doesn't matter. Once you have a spoken-circuit encoding the immediate dynamics of motivations (batteries in the circuit) and how they relate back to ground (resistors and inductors encoding a simplified simulation of a scenario), you can start modeling these dynamics as a literal analog electrical circuit.
In the special case of one power source in the circuit with enough voltage to overwhelm any other, this system of analog simulation should reduce to the ordinary semantics of True/False reasoning. The big voltage represents tautology. If A implies B, and A is connected to the tautology voltage, B is now at the tautology voltage.
All sorts of semantics and reasoning calculi can be encoded in this way. Its a powerful paradigm. Does it actually happen in the brain though? I have no idea.
I realize in retrospect it sounded like I was suggesting absolute dictionary word to single neuron translation. That's not what I meant. What I'm really suggesting is more like a wetware FPGA that does abstract reasoning about people/causation with analog circuit models. It so happens those circuits can be encoded into a spoken transfer format in a very literal way.
Now that I've elaborated further, is that at least developed enough to discuss?
>Your question is about internal representation, but skips the process, and is about a representation that isn't the end of the process either.
I think you missed the essence of my conjecture. I'm not "skipping the process". I'm speculating that there is surprisingly little process to be done at all because language is "close to the metal" (to borrow an analogy).
Here's a simple but limited language. We can have electronic components like batteries and resistors and inductors as our basic noun objects, and our relational verb words are all just telling us if subjects and objects are connected in parallel or series. The only adjectives are numerical pre-fixes for amounts of voltage/resistance/whatever. A "." will suffice as a grammatical particle to mark "end of object". Its a bit verbose and cumbersome, but I can describe whatever circuit I want in these terms. In fact, the parse tree of this grammar can literally encode any loop-free circuit, with sibling branches in the parse tree corresponding to parallel connections and nested branches in the tree corresponding to series connections. I could literally use this grammar and vocab as a file format for an FPGA if I wanted to. One doesn't even need a fully general computation or additional reasoning abilities to "understand" this circuit language. A machine that physically builds the connections according to the grammatical rules as the words stream in would suffice. The semantics of this toy language don't require any further processing or translation.
Now here's the toy brain I envision. Purpose follows a tree structure. Things happen and things exist because someone wills them to be. When you get paid to work, the work serves the will of the company. When you do it for yourself, its your free will. When your computer does something, its because of your will (never its own free will). Each person is both a source of will power and a thing that exists as a matter of ground truth. Thus we can think of a person as a battery in a simplified analog model of our world.
For example you can describe a short circuit of one of these person-batteries with a mutually recursive circuit-statement: "I am because I want to be. I want to be because I am." This is a vacuous truth. You can also make more sophisticated paths from will power (want) to common ground (facts) as a way to encode a more complex scenario. All you have to do is put intermediate things in the path between "I want" and "I am". "I want [A] because [B] so that [C] or [D] which leads to [E] ... so that I continue to be." The full semantics of the objects in the broader context may or may not be known, but it doesn't matter. Once you have a spoken-circuit encoding the immediate dynamics of motivations (batteries in the circuit) and how they relate back to ground (resistors and inductors encoding a simplified simulation of a scenario), you can start modeling these dynamics as a literal analog electrical circuit.
In the special case of one power source in the circuit with enough voltage to overwhelm any other, this system of analog simulation should reduce to the ordinary semantics of True/False reasoning. The big voltage represents tautology. If A implies B, and A is connected to the tautology voltage, B is now at the tautology voltage.
All sorts of semantics and reasoning calculi can be encoded in this way. Its a powerful paradigm. Does it actually happen in the brain though? I have no idea.
I realize in retrospect it sounded like I was suggesting absolute dictionary word to single neuron translation. That's not what I meant. What I'm really suggesting is more like a wetware FPGA that does abstract reasoning about people/causation with analog circuit models. It so happens those circuits can be encoded into a spoken transfer format in a very literal way.
Now that I've elaborated further, is that at least developed enough to discuss?
My suggestion, try the book "realism and reason." It's a collection of papers by Hilary Putnam. It gets into this topic in several different ways, surveys some of the other thinking on it, and so on.
It would be interesting to try to map this tree with a language model like GPT.
For anyone interested in actual neurocomputational theories I can highly recommend the work of Friedemann Pulvermueller.
https://www.sciencedirect.com/science/article/pii/S030100821...
https://www.sciencedirect.com/science/article/pii/S030100821...
This seems very consistent with An embodied grammar of words[1] a less well-known but very compelling pair of theories (Operator Grammar and Perceptual Control Theory).
[1] https://www.researchgate.net/publication/324362482_An_embodi...
[1] https://www.researchgate.net/publication/324362482_An_embodi...
Protip/shameless plug: the MIT Press website now includes a reading enhancement feature called BeeLine Reader, available under the 'Aa' icon. Disclosure: I'm the founder.
Do the changes in color actually reflect the importance of the content? Because if not, I find myself focusing on the wrong things.
Giving that you are a founder, how much money can be made with such a thing?
Giving that you are a founder, how much money can be made with such a thing?
The color changes are just based on relative line position — the most saturated colors are at the line end and beginning. If the colors are distracting, try a different color scheme — the 'Dark' and 'Gray' are fairly subtle.
In terms of the business, BeeLine has browser plugins that bring in decent revenue, but the vast majority comes from licensing to education platforms like Blackboard. Our value prop is two-pronged: we increase accessibility for people with disabilities, while simultaneously increasing time-on-site, free trial conversion, and other KPIs.
Fun fact: BeeLine got its start on HN, way back in the day: https://news.ycombinator.com/item?id=6335784
In terms of the business, BeeLine has browser plugins that bring in decent revenue, but the vast majority comes from licensing to education platforms like Blackboard. Our value prop is two-pronged: we increase accessibility for people with disabilities, while simultaneously increasing time-on-site, free trial conversion, and other KPIs.
Fun fact: BeeLine got its start on HN, way back in the day: https://news.ycombinator.com/item?id=6335784
Purpose and causation (and reasoning therein) is structured like a tree. You can always take a statement and ask for pointers up and down the tree by asking "how" and "why". Ignoring the specific semantics, everything can be modeled as "things" and force in support/opposition from those things to other things. Thus you arrive at a model of whatever semantics which is just networks of lines of support and attack.
So here's my strange idea: what if language maps literally to connections of neurons. The parse tree of a sentence literally encodes the connections of the electrical circuit which models things applying force in support or opposition to other things.
This is probably wrong, but its not obvious why it is wrong, and the process of showing that its wrong probably has some insights into a model that isn't wrong. So I throw my wild conjecture to the wind anyway.