cover friedemann pulvermuller

Friedemann Pulvermuller on word meaning and embodied semantics

  • cover play_arrow

    PLAY EPISODE


cover friedemann pulvermuller
Season 2012
Season 2012
Description arrow_drop_down

Description

Where in the brain does the meaning of a word live, and why does hearing “kick” activate your leg motor cortex? Friedemann Pulvermuller unpacks how the brain grounds language in sensory and motor experience.

Subscribe for more from the Convergent Science Network podcast series.

Friedemann Pulvermuller presents a neurobiological account of word meaning that challenges traditional modular theories of semantics. Drawing on his mentor Valentino Braitenberg’s vision of the cortex as an information mixing system, Pulvermuller argues that meaning arises from distributed cortical circuits where neurons that were originally specialized for vision or motor control become cross-modal through mutual linkage. The result is that understanding a word like “grasp” activates hand motor representations, while “kick” engages leg-related cortical areas, with activation patterns overlapping those produced by actual movements.

The conversation carefully distinguishes four facets of semantics: referential, combinatorial, abstract, and emotional. Referential semantics connects words to objects and actions in the world, solving the symbol grounding problem that purely symbolic approaches cannot address. Combinatorial semantics captures statistical co-occurrence patterns between words, allowing even a blind person to learn that strawberries are red. Abstract semantics, illustrated through the concept of freedom, requires more computational power because multiple diverse prototypical instantiations must be linked through logical either-or operations. Pulvermuller acknowledges these categories represent extremes on a continuum rather than hard boundaries.

The empirical evidence builds from early EEG studies showing differential hemispheric activation for content versus function words, through PET studies of tool and animal naming, to the critical finding that action verbs related to different body parts produce somatotopically organized activation in motor cortex. This body-part specificity, controlled for linguistic confounds like imageability and grammatical class, provided the strongest evidence that semantic processing engages sensorimotor systems in a content-specific manner.

Pulvermuller frames his approach within a Braitenberg-inspired correlation learning framework, where Hebbian strengthening of connections between co-active neural populations creates the distributed circuits that carry meaning, offering a mechanistic bridge between neural anatomy and the richness of human language.

Tagged as:

About the author call_made

CSN Podcasts

Both the triumphs of humanity and its most evil deeds have resulted from collaboration. In a time where humanity is required to aspire to the former and minimize the latter, the question arises of how collaboration arises and why it fails. Surprisingly, this phenomenon, so central to who we are, is not well understood. Hence, a collaborative effort is required to understand collaboration in its full biological, psychological, sociological, cultural, and economic complexity and to translate this understanding into operational impact. This series of podcasts is one step toward achieving these complementary goals. The Collaboration Podcast presents interviews with people who are central orchestrators of collaboration in various domains including business, government, science, art, health, sustainability, and the military. The discussions were conducted by Prof. Dr. Paul F.M.J. Verschure and members of the Program Advisory Committee of the Ernst Strungmann Forum on Collaboration (https://www.esforum.de/forums/ESF32_Collaboration.html) during 2021 and had the goal to sketch a map of opportunities, challenges, and obstacles in human collaboration. The forum took place in May 2022, and now we would like to share this series of interviews with a broader audience. The full report of the Forum will be published in 2023 by MIT Press. The podcast was produced by the Convergent Science Network (https://www.convergentsciencenetwork.org/). Context: The stability of social systems depends critically on realizing sustainable methods of “collaboration,” yet how and by which means collaboration is achieved is not clearly understood; neither are the conditions or processes that lead to its breakdown or failure. Collaboration can be understood as cooperation between agents toward mutually constructed goals. Part of the reason for our lack of understanding is that the phenomenon of collaboration is, by nature, a highly multidisciplinary problem, and effective research into its complexities has been difficult to achieve across the broad range of scientific and technical disciplines involved. The need for a fundamental understanding of collaboration, however, has become increasingly important. Not only does humankind demand answers as it attempts to address critical challenges at multiple scales (e.g., climate change, migration, enhanced automation, social and economic inequality), but ever-increasing technological and economic means of interconnecting people and societies are disrupting long-established, familiar patterns of how we interact. Radical technological changes that are ongoing have the potential to reshape collaboration in ways that are currently hard to predict or influence (e.g., by altering configurations in interaction, information creation, and modes of communication). On one hand, such changes could disrupt hitherto stable forms of collaboration by affecting critical communication channels and traditional roles, as can be observed in the rapidly changing patterns in governance, commerce, and social interaction. Conversely, technology could lead to the emergence of novel, successful forms of collaboration that deviate from traditional “hierarchical” architectures. Evidence of this can be seen in areas as diverse as highly automated manufacturing plants, the open science movement, collaborative software repositories, user-centered services, and the sharing of economy-based modes of organization. Without a fundamental understanding of the mechanisms, processes, and boundary conditions of collaboration, it is not possible to evaluate or predict which of these possible scenarios are sustainable or even plausible. The Forum “How Collaboration Arises and Why it Fails” (May 8–13, 2022, Location: Frankfurt am Main, Germany) Chairs: Andreas Roepstorff and Paul Verschure Program Advisory Committee: Jenna Bednar, Julia R. Lupp, Bhavani R. Rao , Andreas Roepstorff, Ferdinand von Siemens, and Paul Verschure

More posts

Timestamp

  • fast_forward00:00:03 - This is the Convergent Science Network podcast.
  • fast_forward00:00:08 - Leading researchers in the domain of neuroscience, brain theory and technology
  • fast_forward00:00:13 - are interviewed by Paul Verschoor and Tony Prescott.
  • fast_forward00:00:19 - So this is Paul Verschoor with the Convergent Science Network podcast.
  • fast_forward00:00:24 - And I'm speaking now with Friedemann Pulvermuller, who is also a speaker at
  • fast_forward00:00:29 - our summer school here in Barcelona.
  • fast_forward00:00:32 - And Friedemann, you started with a tribute to Valentino Brattenberg in your talk. Why that?
  • fast_forward00:00:39 - Well, because he's my teacher and mentor, and I learned a lot from him.
  • fast_forward00:00:44 - Especially the question how to
  • fast_forward00:00:47 - address cognitive processes
  • fast_forward00:00:50 - this is from a neuroscience perspective
  • fast_forward00:00:53 - looking for mechanistic answers
  • fast_forward00:00:57 - to cognitive questions this uh
  • fast_forward00:01:00 - general approach i would say i inherited from
  • fast_forward00:01:04 - him but now um brettenberg
  • fast_forward00:01:10 - is an anatomist to a large extent yes okay there was
  • fast_forward00:01:13 - also this incredible project on vehicles and psychology which
  • fast_forward00:01:16 - is really an amazing issue on detour in his
  • fast_forward00:01:19 - in his career with quite an impact right but at
  • fast_forward00:01:22 - heart he was an anatomist but in your own work
  • fast_forward00:01:25 - you're not really doing anatomy anymore well not
  • fast_forward00:01:29 - at the moment i would say now nowadays we have dti
  • fast_forward00:01:32 - and then methods to look at
  • fast_forward00:01:35 - cortical connectivity uh which i'm
  • fast_forward00:01:38 - interested in we haven't published about that but but in principle uh that would
  • fast_forward00:01:43 - be very much in the in the in the range of my inner interests and And there
  • fast_forward00:01:51 - are projects such as we are looking at activation spreading over the cortex.
  • fast_forward00:01:58 - And of course, this reflects to a degree the spreading through long-distance
  • fast_forward00:02:03 - cortical connectivity.
  • fast_forward00:02:06 - And of course, the models we are using to explain these activation spreadings,
  • fast_forward00:02:14 - They are grounded in neuroanatomy,
  • fast_forward00:02:19 - so we read a lot of neuroanatomical work and try to make realistic computer
  • fast_forward00:02:26 - models of the brain and its substructures,
  • fast_forward00:02:30 - like the different areas of motor system and the auditory system with a belt
  • fast_forward00:02:37 - and parabelt and whatever it is.
  • fast_forward00:02:38 - And then also putting in those connections that have been documented neuroanatomically.
  • fast_forward00:02:44 - So neuroanatomy plays a big role, but I'm not a neuroanatomist.
  • fast_forward00:02:49 - Right, okay. But then the other thing that you mentioned in relation to Breitenberg was that.
  • fast_forward00:02:56 - He had this working hypothesis of the brain as an information mixing system.
  • fast_forward00:03:00 - Yes. Okay, so what does it actually really mean?
  • fast_forward00:03:02 - Well, it means that it's a system in which indeed there's specialization in
  • fast_forward00:03:09 - the sense that there's an area where visual information comes in,
  • fast_forward00:03:12 - from which motor activation goes out.
  • fast_forward00:03:15 - But that the actual purpose of the cortex is to link as many as possible areas
  • fast_forward00:03:22 - to each other and to provide information mixing in the sense that the motor neuron, finally,
  • fast_forward00:03:29 - isn't a motor neuron only anymore after the linkage with the visual neuron.
  • fast_forward00:03:37 - But the motor neuron becomes a little bit visual as well as,
  • fast_forward00:03:41 - and in the very same way, by mutual linkage with the motor neuron,
  • fast_forward00:03:45 - the visual neuron may become a little bit motor.
  • fast_forward00:03:49 - So there are distributed cortical circuits that now carry multimodal,
  • fast_forward00:03:55 - cross-modal properties, or if you wish, they become quite abstract. All right.
  • fast_forward00:04:00 - So, if I understand you correctly, this information mixing capability is something
  • fast_forward00:04:05 - you ascribe more to the neocortex, not to the brain overall.
  • fast_forward00:04:09 - Absolutely. The picture I showed was actually a picture of the cortex.
  • fast_forward00:04:15 - The cortex would be the information mixer.
  • fast_forward00:04:18 - The cerebellum would be more a hetero-associative network that just links one
  • fast_forward00:04:27 - pattern to a subsequent pattern.
  • fast_forward00:04:29 - So, the nature of that is very different. Okay.
  • fast_forward00:04:32 - And indeed, also in your own work, you focus very much on these mixing capabilities
  • fast_forward00:04:37 - of the neocortex, in particular with respect to meaning and semantics with respect
  • fast_forward00:04:43 - to language, right? So, word meaning.
  • fast_forward00:04:45 - So, what's the problem really around meaning?
  • fast_forward00:04:49 - Why are you worried about meaning? I'm not so much worried about meaning.
  • fast_forward00:04:53 - Okay. I think meaning is a very straightforward example case,
  • fast_forward00:04:58 - and we can understand aspects of it very easily.
  • fast_forward00:05:05 - But I'm surprised that you say you're not worried about it, because in some
  • fast_forward00:05:10 - sense, also your presentation, right, you try to show how actually in the past,
  • fast_forward00:05:15 - or sort of standard models of meaning are actually insufficient, right?
  • fast_forward00:05:19 - The standard models of meaning are often said, look, if somewhere in the brain
  • fast_forward00:05:22 - there's a meaning module with all sorts of magical capabilities, that's insufficient.
  • fast_forward00:05:28 - So this is really the challenge you try to answer now.
  • fast_forward00:05:32 - So what is wrong with these sort of textbook models of meaning and meaning modules? Yes.
  • fast_forward00:05:38 - Well, one thing that is wrong or insufficient is if I have a meaning module
  • fast_forward00:05:44 - in which the meaning of each word or each concept is defined by relationship
  • fast_forward00:05:50 - to other words and other concepts,
  • fast_forward00:05:53 - then the so-called symbol grounding problem remains unanswered.
  • fast_forward00:06:01 - So if I can use the word
  • fast_forward00:06:04 - red very well in context such as a strawberry is red and lips are red and red
  • fast_forward00:06:17 - symbolizes love or whatever,
  • fast_forward00:06:21 - it's not sufficient.
  • fast_forward00:06:25 - In order to document that I'm able to use the word red, I would need to point
  • fast_forward00:06:32 - to the right color in a certain context.
  • fast_forward00:06:35 - In order to know what a strawberry is, I would also need to find out which of
  • fast_forward00:06:39 - the berries is actually the strawberry.
  • fast_forward00:06:41 - If I don't have this capability, one of the criteria of semantic knowledge is not satisfied.
  • fast_forward00:06:49 - And if I define word meaning just by semantic relationships,
  • fast_forward00:06:57 - relationships between symbols, I do not have this grounding in knowledge of the world.
  • fast_forward00:07:06 - This point has been made by Harnatt, Searle, many scientists. Right, exactly. Yeah.
  • fast_forward00:07:13 - But what does this mean in practical terms? Does it mean that?
  • fast_forward00:07:15 - If we want a theory of meaning, must map it back to interaction with the real world?
  • fast_forward00:07:21 - Or should just map it back to the real world with respect to, let's say, reference?
  • fast_forward00:07:25 - Let's say a strawberry just means that I can point to an object in the world
  • fast_forward00:07:28 - and say, okay, this impression, the sensor states that are triggered by this
  • fast_forward00:07:33 - object, as long as you can bring these together, this is meaning.
  • fast_forward00:07:37 - Or is it really about me physically interacting and acting up on,
  • fast_forward00:07:42 - that's to picking it up and smelling it and tasting it and so on?
  • fast_forward00:07:45 - Is that action component critical or just one among many components that provide
  • fast_forward00:07:50 - statistics on which meaning flows?
  • fast_forward00:07:53 - Well, object reference, of course, meaning that I know to which object my given
  • fast_forward00:08:01 - word like strawberry applies is a necessary component.
  • fast_forward00:08:05 - So for words like strawberry, I want to show the point of the right thing.
  • fast_forward00:08:11 - I want to know more, of course.
  • fast_forward00:08:13 - I want to know how it tastes, how it smells as well.
  • fast_forward00:08:16 - If I'm slightly deprived, if I'm blind, if I cannot taste so well,
  • fast_forward00:08:24 - or if there's a sensory deprivation of different time, of course,
  • fast_forward00:08:27 - I may be limited on one of these dimensions,
  • fast_forward00:08:29 - but there are still the other dimensions, sensory knowledge,
  • fast_forward00:08:36 - so to speak, that is of semantic relevance.
  • fast_forward00:08:41 - There are other kinds of words, such as words with which we speak about actions.
  • fast_forward00:08:48 - For example, a word like grasp, here there's no object to point to.
  • fast_forward00:08:53 - Here there's no object reference in the same sense.
  • fast_forward00:08:57 - We use it to speak about actions, but we also need to have this knowledge in
  • fast_forward00:09:02 - which action contexts the word can be properly applied.
  • fast_forward00:09:05 - And in which contexts it fails to be appropriately used.
  • fast_forward00:09:14 - So it's not just visual perception to which the words relate objects.
  • fast_forward00:09:23 - It's also actions. And for abstract concepts, it's a very similar issue,
  • fast_forward00:09:28 - only possibly slightly more complex. If I want to teach my child what freedom means, what would I do?
  • fast_forward00:09:39 - I would explain him or her that somebody who is living in jail and who is allowed
  • fast_forward00:09:47 - out after several years is now free.
  • fast_forward00:09:50 - And I would say, well, and this guy who is in handcuffs and somebody unlocks
  • fast_forward00:09:57 - them and lets him go, freeze this guy.
  • fast_forward00:10:02 - And I could also say that a judge sitting there and just doing nothing except
  • fast_forward00:10:07 - for saying a sentence, well, you are not guilty and you are free.
  • fast_forward00:10:16 - You do not have to stay in our prison here.
  • fast_forward00:10:19 - So these are different instantiations of the freedom concept, so to speak.
  • fast_forward00:10:25 - And in order to be able to use this abstract term, I would need to know how
  • fast_forward00:10:32 - to apply it in such prototypical circumstances.
  • fast_forward00:10:36 - Of course, now everybody has his or her own experiences with strawberries,
  • fast_forward00:10:41 - with freedom, and so on and so forth.
  • fast_forward00:10:43 - But there is a common knowledge about prototypical situations,
  • fast_forward00:10:47 - both for the abstract and for the concrete items. Okay.
  • fast_forward00:10:51 - So then you're saying meaning as such comes in degrees, let's say.
  • fast_forward00:10:56 - You can tie in more or less modalities and submodalities.
  • fast_forward00:11:00 - Absolutely. Right. And are there some boundaries to that?
  • fast_forward00:11:04 - Is there some boundary? Let's say, does it need to be symbolic in some sense
  • fast_forward00:11:07 - or can also be just purely analog?
  • fast_forward00:11:10 - Does it sort of means how literal should be the connection to,
  • fast_forward00:11:14 - let's say, the neuronal activity that an object might trigger?
  • fast_forward00:11:19 - So are the boundaries on this notion of meaning if it comes in gradations?
  • fast_forward00:11:24 - And now the term symbolic.
  • fast_forward00:11:28 - Is well should probably be
  • fast_forward00:11:31 - explained so may i ask what you how you
  • fast_forward00:11:34 - use this well it's in artificial intelligence
  • fast_forward00:11:37 - it's sometimes a symbolic approach is sometimes that approach that doesn't have
  • fast_forward00:11:42 - the neuronal basis right now of course we are talking neurons and and therefore
  • fast_forward00:11:48 - it doesn't it's not just symbolic it's always with a neural basis so the main
  • fast_forward00:11:52 - question breitenbergian question if you wish,
  • fast_forward00:11:56 - is how can we find a neurobiological underpinning for things such as symbols?
  • fast_forward00:12:03 - So in that sense, it doesn't stay symbolic, but it becomes symbolic with a brain grounding.
  • fast_forward00:12:09 - Right, with sort of an implementation of the physical component.
  • fast_forward00:12:12 - Yes, with a mechanism behind it. Exactly.
  • fast_forward00:12:14 - So with symbolic in the way I intend this also, that's why I mentioned analog,
  • fast_forward00:12:19 - I think this is an important position.
  • fast_forward00:12:22 - Because in the analog case, the object will drive a sensor sheet,
  • fast_forward00:12:28 - sensor sheet will drive neural activity, and it is sort of a direct mapping, right?
  • fast_forward00:12:32 - So the activity is not transformed in some way, but it's really this vector
  • fast_forward00:12:37 - of responses, the state of responses induced by the way the object in the outside
  • fast_forward00:12:42 - world tickles the sensor sheet that gives you now an analog representation.
  • fast_forward00:12:46 - A symbolic representation doesn't have that feature. It's really disconnected
  • fast_forward00:12:50 - from the way in which the outside world is tickling the sensor sheets.
  • fast_forward00:12:54 - So it's a decoupling between an internal representation and how this can be
  • fast_forward00:12:58 - activated by the outside world.
  • fast_forward00:13:01 - Well, I'm not so sure about the latter, as I try to explain in this case of
  • fast_forward00:13:07 - the abstract meaning of freedom or free, of the word free.
  • fast_forward00:13:12 - If we learn the meaning or if we teach, well, the practical example is always
  • fast_forward00:13:18 - related to practical activities like teaching a child what the meaning is. And what would you do?
  • fast_forward00:13:25 - You would present concrete examples, a range of communication contexts where
  • fast_forward00:13:33 - the word is appropriately applied, interaction contexts also.
  • fast_forward00:13:37 - But so there isn't a complete detachment between the meaning and these typical instantiation.
  • fast_forward00:13:46 - One could even claim that the knowledge about the typical instantiation and
  • fast_forward00:13:52 - instantiations is necessary for the knowledge about the symbolic meaning.
  • fast_forward00:14:00 - And I wouldn't go so far that I would say it exhausts the meaning, but it's necessary.
  • fast_forward00:14:08 - And what is different from a case of knowing the meaning of the word strawberry,
  • fast_forward00:14:18 - where there is probably one could argue there's one prototype.
  • fast_forward00:14:22 - A main difference is that in In this freedom case, there are many different prototypes.
  • fast_forward00:14:28 - And therefore, one couldn't say, well, and it's actually this hand movement
  • fast_forward00:14:32 - of unlocking the handcuffs, which relates to the freedom concept.
  • fast_forward00:14:38 - It's actually a vast variety of different actions that could go through as good
  • fast_forward00:14:46 - examples of freeing somebody, a verbal action, a hand action, and so on and so forth.
  • fast_forward00:14:51 - But in the case of a strawberry it's.
  • fast_forward00:14:56 - There could be one prototype only. So we need more computational power for the abstract concept.
  • fast_forward00:15:04 - We need to calculate something like an either-or function over different instantiations.
  • fast_forward00:15:10 - And therefore, as you correctly mentioned,
  • fast_forward00:15:14 - there's a degree of detachment between the instantiation, the concrete situational
  • fast_forward00:15:24 - context, prototypical situational context, because there are so many of them.
  • fast_forward00:15:28 - By the way, the correlation rule captures that very well. If I correlate two
  • fast_forward00:15:35 - things with each other, then there may be a link.
  • fast_forward00:15:40 - But if I try to store the knowledge that one thing can have 10 different corresponding instantiations,
  • fast_forward00:15:51 - then the correlation rule would actually produce a little bit of a link for each.
  • fast_forward00:16:00 - But also, if I learn that A and B belong together, there's strengthening of the connection.
  • fast_forward00:16:07 - But if I learn that there's A and C that also belong together,
  • fast_forward00:16:10 - there's a little bit of weakening. Yeah, but now we're jumping forward, right?
  • fast_forward00:16:13 - This is also very much related to the model that you built of meaning.
  • fast_forward00:16:17 - And before we get to the model, I would like to first look at sort of the data
  • fast_forward00:16:22 - you have collected on meaning in the brain.
  • fast_forward00:16:25 - And before getting to the data, another definitional issue that also came up
  • fast_forward00:16:30 - this morning and would be useful to, I think, get it out there,
  • fast_forward00:16:34 - was this whole issue about, again, but what then is the difference between meaning
  • fast_forward00:16:39 - and semantics and the notion of a concept?
  • fast_forward00:16:42 - But why not just speak of concepts? Why not say, look, if it's just about,
  • fast_forward00:16:46 - let's say, the kind of information in the outside world.
  • fast_forward00:16:51 - That co-occurs, right, given a certain label, a word, why not call that just a concept, right?
  • fast_forward00:17:00 - That's just a cloud of points in some high-dimensional space,
  • fast_forward00:17:02 - so they belong together because they co-occur, so I call it a concept.
  • fast_forward00:17:05 - But when do we call that a semantic meaning as opposed to just a concept?
  • fast_forward00:17:12 - Well, in semantic theory, it's
  • fast_forward00:17:15 - just conventional to distinguish between concepts and semantics of words.
  • fast_forward00:17:22 - And the very simple difference is that one could say that semantics.
  • fast_forward00:17:30 - The meaning of a word, semantics is the same thing as meaning,
  • fast_forward00:17:34 - and the meaning of a word is the concept with a regular relationship to a word form or a symbol.
  • fast_forward00:17:41 - So it's just, if you wish, a subset of the space of concepts,
  • fast_forward00:17:46 - but those that have a clear relationship to words established in the language community.
  • fast_forward00:17:54 - So if I personally have a bad experience with fish and therefore would have
  • fast_forward00:18:02 - a, well, always a startle response when seeing or tasting one.
  • fast_forward00:18:08 - The word fish wouldn't have the meaning of being something bad because it's just my own experience.
  • fast_forward00:18:16 - So that the the word semantics would
  • fast_forward00:18:19 - always have this language community aspect so
  • fast_forward00:18:23 - it should be shared by many right otherwise in order to allow that what i say
  • fast_forward00:18:28 - is understood by others in the same way so if my personal experience is slightly
  • fast_forward00:18:33 - different from the from the from the from what is established in the in the
  • fast_forward00:18:38 - language community then one would,
  • fast_forward00:18:40 - It would need to be discussed whether this aspect should belong to a semantic space, so to speak.
  • fast_forward00:18:47 - So the concepts basically define, let's say, the co-occurrences of certain statistical
  • fast_forward00:18:55 - states out there in the world.
  • fast_forward00:18:56 - But as soon as these start to co-occur together with words, you call it semantics. Yes.
  • fast_forward00:19:02 - But it also would mean that your theory in principle would generalize to the notion of concepts.
  • fast_forward00:19:08 - Absolutely. And you use the semantic linguistic case more as your test case
  • fast_forward00:19:12 - in some sense to get access to this conceptual space.
  • fast_forward00:19:16 - Also, for the case of language, I think for me this is much simpler because
  • fast_forward00:19:22 - asking what is the meaning of the word can be very closely related to the question,
  • fast_forward00:19:28 - how do I teach the meaning of this word, for example, to a language learner, a child.
  • fast_forward00:19:32 - However, the question about the nature of concepts is much more nebulous.
  • fast_forward00:19:37 - So, I can say, well, and it's very easy to say, well, concepts are inborn.
  • fast_forward00:19:43 - They are just in our heads by nature, by natural laws and genetics.
  • fast_forward00:19:50 - And I don't want to know about it.
  • fast_forward00:19:53 - I just postulate it. However, if I do this for semantics, then I run into a problem.
  • fast_forward00:19:58 - Because I assume I have an inborn knowledge of stinginess.
  • fast_forward00:20:10 - Now, how would I become able to link the word form stinginess,
  • fast_forward00:20:18 - the word stinginess, to this particular inborn concept?
  • fast_forward00:20:22 - So the semantic learning would require that stinginess would become manifest
  • fast_forward00:20:28 - somehow, otherwise I couldn't teach it.
  • fast_forward00:20:31 - So therefore, the semantic case is much simpler and much easier to address than
  • fast_forward00:20:37 - the conceptual case. If I come up with a conceptual theory, I can stay in nebulous space.
  • fast_forward00:20:44 - But apparently you did figure out I'm Dutch.
  • fast_forward00:20:48 - That's why I took stinginess as your example. No, no, I'm Swabian,
  • fast_forward00:20:52 - and people in Swabia, southwest Germany, they are said to be very stingy.
  • fast_forward00:20:59 - Really? Oh, the Dutch is the same. Okay.
  • fast_forward00:21:01 - So we are innately stingy. Yes. Okay.
  • fast_forward00:21:05 - So there must be a genetic relationship between Swabians, Dutch,
  • fast_forward00:21:09 - and Scotsman. That's going to be our next collaborative project.
  • fast_forward00:21:13 - Okay, so now we did the preliminaries, right? So the definitions are out of the way.
  • fast_forward00:21:18 - Yeah. We look now at, now we can move on to this notion of the mechanisms of
  • fast_forward00:21:23 - meaning, because that will bring you to the brain, right?
  • fast_forward00:21:26 - And there you distinguish that's referential semantics, abstract semantics,
  • fast_forward00:21:31 - emotional semantics, and combinatorial semantics.
  • fast_forward00:21:34 - Yes. Why these four? Why these four?
  • fast_forward00:21:41 - Well, the case of referential semantics we have discussed, or if there's a link
  • fast_forward00:21:47 - between words and the world, or even construction sentences and the world.
  • fast_forward00:21:54 - We speak about things, about actions, about interactions, and this knowledge
  • fast_forward00:22:01 - of this relationship has to do with the meaning of the terms. of the terms.
  • fast_forward00:22:07 - Then there's this claim that there's a relationship among the symbols.
  • fast_forward00:22:13 - And this is to a degree relevant for the meaning.
  • fast_forward00:22:17 - And there's no doubt that this idea is correct.
  • fast_forward00:22:21 - For example, a blind person can learn that strawberries are red,
  • fast_forward00:22:26 - not because this person has ever seen a red strawberry,
  • fast_forward00:22:31 - but simply because the word red and the word strawberry frequently co-occur in sentences.
  • fast_forward00:22:36 - And it's not sentences that also have a not in them.
  • fast_forward00:22:41 - So there's some combinatorial data around that also allows us to draw conclusions
  • fast_forward00:22:50 - that are semantic in nature,
  • fast_forward00:22:53 - in the sense that we can deduce semantic statements,
  • fast_forward00:22:58 - of the thought strawberry has to do something Something to do with red or is
  • fast_forward00:23:05 - red, if I know that red is a feature of objects.
  • fast_forward00:23:12 - Now, this is, and of course we know that words occur together,
  • fast_forward00:23:18 - other words occur together not so frequently.
  • fast_forward00:23:22 - There's a correlation not only between words and the world. So I use the word
  • fast_forward00:23:28 - to speak about this type of object, and therefore the object may co-occur with
  • fast_forward00:23:34 - the occurrence of the word form, or I would...
  • fast_forward00:23:38 - It doesn't require that the strawberry is indeed present in the environment
  • fast_forward00:23:42 - while I speak. I could use the word strawberry.
  • fast_forward00:23:45 - I could also imagine, think of the object and at the same time use the word,
  • fast_forward00:23:51 - so simulate the object or the scene.
  • fast_forward00:23:57 - And this could lead to a semantic linkage of a referential type.
  • fast_forward00:24:02 - But then the co-occurrence of the words in strings would also trigger similar processes.
  • fast_forward00:24:09 - We know that nerve cells in the cortex strengthen their links when they are active together.
  • fast_forward00:24:16 - And assume that there's a neural population that corresponds to my word number
  • fast_forward00:24:22 - one, red, and my word number two, strawberry.
  • fast_forward00:24:25 - If they occur together very frequently, then the connections between them,
  • fast_forward00:24:30 - if any, would strengthen.
  • fast_forward00:24:31 - And if they, however, if they appear in different contexts, each on its own,
  • fast_forward00:24:39 - then this connection might weaken again.
  • fast_forward00:24:42 - So there is a degree of mapping also of word co-occurrences,
  • fast_forward00:24:49 - and this leads to the postulate that we have this storage of combinatorial semantics,
  • fast_forward00:24:58 - which complements the semantic knowledge that relates to the world.
  • fast_forward00:25:05 - So, there's this word-world relationship, but also this word-word relationship
  • fast_forward00:25:12 - that is easily mapped by a correlation storage device, such as our cortex.
  • fast_forward00:25:19 - So, over the years, you have actually accumulated a lot of evidence for this hypothesis, right?
  • fast_forward00:25:27 - So what do you see as the outstanding pieces of data that support this view on semantics?
  • fast_forward00:25:38 - If you allow me, your previous question was actually aiming at two more facets.
  • fast_forward00:25:46 - Yes. One about abstract semantics, one about emotional semantics. Yes, exactly.
  • fast_forward00:25:51 - And maybe I could very briefly address those two too.
  • fast_forward00:25:56 - Okay. So the abstract semantics is special in the sense of the freedom example
  • fast_forward00:26:04 - I started our discussion with. I presented it at the start of our discussion.
  • fast_forward00:26:10 - So the strawberry example is very simple.
  • fast_forward00:26:15 - One object scheme, one word, linked together. Freedom, much more difficult semantically.
  • fast_forward00:26:23 - Very different prototypical instantiations of the meaning of that concept,
  • fast_forward00:26:29 - of that many meaning facets, if you wish. and there needs to be more computational power.
  • fast_forward00:26:38 - There needs to be a neural element
  • fast_forward00:26:41 - that says it can be this or it can be this or that or that or that.
  • fast_forward00:26:46 - So a list of prototypes so to speak and an either or or just an or connection between them.
  • fast_forward00:26:53 - And this is the special thing that in this case such or or either or connections
  • fast_forward00:27:00 - links would need to come in So a little bit more, if you wish,
  • fast_forward00:27:05 - symbolic neuronal mechanisms.
  • fast_forward00:27:09 - But we have no problems with these mechanisms because this is actually what
  • fast_forward00:27:14 - nerve cells are made for, to do such computations.
  • fast_forward00:27:17 - One reason why I skipped over it to fill that in further and to go to the neural
  • fast_forward00:27:23 - mechanism was because I wanted to challenge you later on these distinctions
  • fast_forward00:27:30 - you're making. Maybe we can do that now.
  • fast_forward00:27:32 - Because in some sense, I could argue, look, let's take referential semantics.
  • fast_forward00:27:37 - Okay, so we take headphone.
  • fast_forward00:27:39 - Okay, so I see a headphone out there in the world, but the headphone in itself
  • fast_forward00:27:43 - is comprised of many components.
  • fast_forward00:27:45 - And certainly if I look at the headphone as just a visual stimulus.
  • fast_forward00:27:50 - It is completely fragmented in my visual cortex.
  • fast_forward00:27:53 - When it enters my V1, it's just a massive puzzle with millions of pieces that
  • fast_forward00:27:59 - I have to sort of actively reassemble.
  • fast_forward00:28:01 - So in some sense, at that level of now visual meaning, meaning I'm already solving
  • fast_forward00:28:07 - a combinatorial problem, right?
  • fast_forward00:28:09 - So in some sense, this raises this issue about the distinctions you're making
  • fast_forward00:28:12 - here and whether they're really a minimal interpretation, right?
  • fast_forward00:28:18 - I could say, well, maybe combinatorial is subsumed also in referential.
  • fast_forward00:28:23 - So this is not really a clear distinction.
  • fast_forward00:28:25 - And I could possibly make the same argument with abstraction because to go from
  • fast_forward00:28:30 - the headphone input states of my primary visual cortex text to my concept headphone,
  • fast_forward00:28:37 - so before it reaches a semantic stage.
  • fast_forward00:28:40 - Now, just the integration of all this information that floats around in my brain
  • fast_forward00:28:44 - around headphone, also that is a form of abstraction.
  • fast_forward00:28:47 - It must be abstracted. I cannot rely on what's out there in the world.
  • fast_forward00:28:50 - So in some sense, this distinction you make might be a little bit arbitrary.
  • fast_forward00:28:56 - There might not be hard borders between them.
  • fast_forward00:28:59 - Indeed. I think this criticism is, to a degree, appropriate.
  • fast_forward00:29:05 - So there are different types of chairs, different types of headphones,
  • fast_forward00:29:09 - and if we speak about headphones, we might want to include the more or less prototypical ones.
  • fast_forward00:29:16 - And, of course, if we speak about
  • fast_forward00:29:18 - animals, there's a long list of possible referent objects and objects.
  • fast_forward00:29:29 - I agree that they are not just the extremes of the spectrum.
  • fast_forward00:29:35 - So an item such as a fruit, which comes with very little variation,
  • fast_forward00:29:45 - even though if you think of strawberries, they can be very small,
  • fast_forward00:29:50 - very large, so a degree of abstraction is necessary there too.
  • fast_forward00:29:54 - And on the other end of the scale, there's an abstract concept where there's
  • fast_forward00:30:00 - really such a wide range of different instantiations that there's no way to
  • fast_forward00:30:08 - achieve much without such logical operations.
  • fast_forward00:30:12 - In between, there are cases of larger category terms or then of terms,
  • fast_forward00:30:18 - as you said, like headphones,
  • fast_forward00:30:20 - which can come in very different shapes where a degree of either-or computation
  • fast_forward00:30:26 - might still be necessary.
  • fast_forward00:30:29 - At a at a basic level category right exactly no yeah but i i agree so there's
  • fast_forward00:30:35 - no clear boundary here this is entire there they are the extreme cases but there's
  • fast_forward00:30:42 - a lot right lots of material in between so you could say ontologically uh in
  • fast_forward00:30:47 - terms of what's really going on in the brain.
  • fast_forward00:30:50 - It might be more a diffuse and a continuous process, but as a research heuristic,
  • fast_forward00:30:55 - it could still help you to actually get access to this, right?
  • fast_forward00:30:58 - So this is maybe how we should look at that.
  • fast_forward00:31:01 - And what we do is we then ask our experimental subjects and ask them 150 questions
  • fast_forward00:31:09 - about the aspects of the meaning of words.
  • fast_forward00:31:12 - And we pick out some of those words with very simple semantics where they say,
  • fast_forward00:31:18 - for example, this word relates to actions I typically perform with my hand, for example.
  • fast_forward00:31:23 - And of course, the action verb to free is not among those.
  • fast_forward00:31:31 - And same with object words as well.
  • fast_forward00:31:35 - And if they are then words with very variable semantics, then those would fall
  • fast_forward00:31:44 - out of these experiments. Right, exactly.
  • fast_forward00:31:48 - Okay, so, but I think we got agreement here. This is very good.
  • fast_forward00:31:51 - Because now when you can go back to the follow-up question, which was,
  • fast_forward00:31:55 - okay, but where's the data now, right?
  • fast_forward00:31:56 - Which is, where is the data that would support this interpretation of semantics?
  • fast_forward00:32:03 - Yes. So where should we start there? What was the first observation,
  • fast_forward00:32:08 - let's say, that gave you hope that this more, let's say, statistical interpretation
  • fast_forward00:32:13 - of word meaning would actually pan out? That's really what the brain is relying on.
  • fast_forward00:32:19 - Sorry, again, this is now about statistical. No, no, wait, wait. I just want to go back.
  • fast_forward00:32:24 - So let's go back again now to the referential semantics.
  • fast_forward00:32:27 - Yes. So we just go to the case where we have word meaning with respect to objects
  • fast_forward00:32:33 - out there in the world, right? The direct reference.
  • fast_forward00:32:35 - And this direct reference, the word meaning is now, there is occurring because
  • fast_forward00:32:40 - aspects of this object co-occur systematically.
  • fast_forward00:32:46 - Right? So statistically, there's a pattern. The brain can pick up this pattern.
  • fast_forward00:32:50 - And now, you know, Friedemann Pulvermuller, seeing this from the outside,
  • fast_forward00:32:54 - says, aha, here is word meaning. Right?
  • fast_forward00:32:57 - So what were these pieces of data? What were the observations that allowed you to say that?
  • fast_forward00:33:02 - Yeah. Well, if you ask me about the history of it or what gave me hope for the
  • fast_forward00:33:07 - first time, then I should say, well, reading a paper by Helen Neville from 1992.
  • fast_forward00:33:15 - 1992, and this was about words related to grammar, function,
  • fast_forward00:33:22 - grammatical function words, and content words, nouns and verbs mainly.
  • fast_forward00:33:27 - And they were presented in an
  • fast_forward00:33:29 - EEG experiment, and there was a degree of activation of both hemispheres,
  • fast_forward00:33:34 - and it was EEG recordings, and there was a lot of activation on both sides of
  • fast_forward00:33:41 - the brain, while for these grammatical
  • fast_forward00:33:44 - words, there was just a localised activation in the left hemisphere.
  • fast_forward00:33:51 - And this is of course compatible with the view that these content words,
  • fast_forward00:33:55 - these action verbs and object nouns and maybe some other items as well,
  • fast_forward00:34:01 - they activate a lot of the.
  • fast_forward00:34:04 - A lot of semantic links in various cortical areas, visual cortex, maybe motor cortex.
  • fast_forward00:34:10 - This is more speculation than data. The data would actually tell you,
  • fast_forward00:34:13 - but it fitted nicely into the model.
  • fast_forward00:34:17 - And we then went on and learned to use EEG by then and replicated this experiment
  • fast_forward00:34:25 - now with better matching of the stimuli in order to exclude a range of confounds.
  • fast_forward00:34:30 - Looked that these words were exactly matched for a range of psycholinguistic
  • fast_forward00:34:35 - features which we now know influence the brain response very much.
  • fast_forward00:34:40 - And we could confirm that there's a strong laterality for the grammatical items
  • fast_forward00:34:45 - and the whole brain, or there's a brain activation pattern compatible with whole
  • fast_forward00:34:51 - cortex activation for these more meaningful items.
  • fast_forward00:34:55 - Now, the whole series of experiments is, of course, entirely insufficient to
  • fast_forward00:34:59 - address the question, but you asked what gave me hope. Yeah, exactly.
  • fast_forward00:35:04 - Because this kind of experiment was heavily confounded. Function words are grammatical items.
  • fast_forward00:35:11 - There's a lot of syntactic knowledge attached to them.
  • fast_forward00:35:14 - Now the content words, they have meaning, but they are combinatorially very different.
  • fast_forward00:35:20 - They are imageable. They give rise to secondary processes, cognitive processes
  • fast_forward00:35:29 - like imagination, function, words, tone, and so on and so forth.
  • fast_forward00:35:33 - And we could match some basic psycholinguistic variables,
  • fast_forward00:35:37 - as I said, and the differences persisted, but still it's not the best way of
  • fast_forward00:35:45 - addressing the semantic questions.
  • fast_forward00:35:48 - And the next thing that gave me hope was work on word category processing or
  • fast_forward00:35:57 - category-specific processes.
  • fast_forward00:35:59 - For example, Alex Martin at NIH did experiments on naming,
  • fast_forward00:36:05 - tool naming and animal naming, and found differential activation
  • fast_forward00:36:08 - at that point with PET and a little bit of motor cortex activation already then
  • fast_forward00:36:15 - for the tool words and a lot of visual and temporal activation for the animal word naming.
  • fast_forward00:36:26 - Now, the problem here is, of course, that you start with pictures and you don't
  • fast_forward00:36:31 - know to what degree the physical features of the pictures influence this and
  • fast_forward00:36:37 - to what degree the picture processing invokes some differential activation or
  • fast_forward00:36:42 - whether it's language and concept related.
  • fast_forward00:36:46 - And we went on to look at nouns and verbs.
  • fast_forward00:36:50 - Nouns and verbs activated. If we take concrete action verbs and object nouns,
  • fast_forward00:36:57 - there's a difference of activation.
  • fast_forward00:36:59 - But of course, also here, linguistic confounds come in. So all of this work
  • fast_forward00:37:04 - gave us a lot of hope, but it wasn't the final answer to the question.
  • fast_forward00:37:08 - I think the first time we were really convinced that now we have something in
  • fast_forward00:37:13 - hand to postulate that it's semantics and only semantics that is reflected in local brain response.
  • fast_forward00:37:19 - And that this local brain response could also be related to activations that
  • fast_forward00:37:25 - have to do with action, with perception, was when we looked at different action patterns.
  • fast_forward00:37:33 - Verbs, actually, action verbs that relate to different parts of the body.
  • fast_forward00:37:37 - And then we found, for example, that words such as grasp, actions you perform
  • fast_forward00:37:42 - with the hand, would activate the hand motor representation,
  • fast_forward00:37:48 - actually overlapping with those areas in those experimental subjects that were
  • fast_forward00:37:54 - also activated when they actually moved their finger or some upper extremity.
  • fast_forward00:37:59 - And the same thing we found for leg-related words like kick or walk,
  • fast_forward00:38:05 - and they activated regions overlapping now with those regions that were also
  • fast_forward00:38:12 - activated when people were tapping their foot or something, moving the foot.
  • fast_forward00:38:18 - And that gave us, because here the linguistic confounds had been ruled out,
  • fast_forward00:38:24 - so the words were similar, also according to grammatical and whatever other possibilities.
  • fast_forward00:38:29 - They were equally imagable, action-related, and so on and so forth,
  • fast_forward00:38:33 - but there was a different body-part relationship.
  • fast_forward00:38:36 - Now, we wouldn't claim, and we haven't actually, that the motor system here
  • fast_forward00:38:41 - houses semantics and only the motor system.
  • fast_forward00:38:44 - But we would say that by activation in the motor system, some aspects of the
  • fast_forward00:38:50 - meaning of these words are reflected.
  • fast_forward00:38:53 - And I think this has been replicated in many studies since, even if there's
  • fast_forward00:39:02 - a degree of variability of these motor activations as a function of context,
  • fast_forward00:39:06 - as a function of task, and interestingly.
  • fast_forward00:39:10 - As a function of the communicative function the words have in a particular situation.
  • fast_forward00:39:16 - Situation, but I think it's generally it could be confirmed and there is evidence that words that.
  • fast_forward00:39:27 - That semantically relate to actions, involve to a degree the motor system.
  • fast_forward00:39:32 - Right. But now, would you see something similar for words that invoke visual perception?
  • fast_forward00:39:38 - Absolutely, yeah. We did very similar experiments now for color and shape words,
  • fast_forward00:39:45 - where there was a similar double dissociation in the visual system.
  • fast_forward00:39:49 - Right. So it's a very general feature that you observe. And there are other
  • fast_forward00:39:55 - groups who have done similar work.
  • fast_forward00:39:59 - For example, Bartholow and Simmons, they looked at color words and those areas
  • fast_forward00:40:09 - that are especially important for color processing.
  • fast_forward00:40:13 - And there was an overlap of activations. And Markus Kiefer in Ulm, Germany,
  • fast_forward00:40:19 - he looked at sound-related words, and a range of people here in the vicinity of Barcelona,
  • fast_forward00:40:31 - they looked very closely at other modality words, such as odor and taste words.
  • fast_forward00:40:42 - So, with respect to this referential semantics, the first observation was,
  • fast_forward00:40:52 - look, action allows you to interpret the response, right?
  • fast_forward00:40:55 - Because you could see how these responses were invading motor execution systems. Yes.
  • fast_forward00:41:01 - But now this raises a number of questions, right? But one is,
  • fast_forward00:41:04 - okay, but then now the brain is mapping all these inputs into a conceptual space.
  • fast_forward00:41:12 - And this conceptual space has some dimensionality because now you're saying,
  • fast_forward00:41:15 - well, it might go towards motor execution systems.
  • fast_forward00:41:18 - It could go towards vision, audition, affection, taste.
  • fast_forward00:41:23 - But is that the intrinsic dimensionality of this semantic space in your mind?
  • fast_forward00:41:28 - Or is that not how it's organized?
  • fast_forward00:41:32 - Well, I would say these modality dimensions are certainly part,
  • fast_forward00:41:41 - as I would construe it, part of the semantic space.
  • fast_forward00:41:45 - They wouldn't exhaust the semantic space.
  • fast_forward00:41:48 - I wouldn't… So what other dimensions are there?
  • fast_forward00:41:54 - If it doesn't exhaust the space, then there must be other aspects to its organization
  • fast_forward00:41:58 - that it doesn't capture.
  • fast_forward00:42:01 - One aspect we skipped entirely, this was emotion relationship.
  • fast_forward00:42:06 - Of course, as we can speak about actions we do with our body,
  • fast_forward00:42:12 - we can speak about objects in the world.
  • fast_forward00:42:15 - In a very similar sense, we can speak about entities that are,
  • fast_forward00:42:22 - in some views, enclosed in our body, internal states, as they have been called.
  • fast_forward00:42:30 - Like if I speak about happiness or joy, this is nothing that would be an object
  • fast_forward00:42:36 - in the world or nothing, and not an action either.
  • fast_forward00:42:39 - So how would that work? And of course, we link the meaning of these items to
  • fast_forward00:42:45 - activation in the limbic system.
  • fast_forward00:42:48 - So it's probably limbic circuits and basal ganglia activation,
  • fast_forward00:42:56 - amygdala and cingulum activation, anterior cingulum, and some parts of the insula play a role.
  • fast_forward00:43:05 - So, we have this corticolimbic linkage here for internal state emotion words especially.
  • fast_forward00:43:20 - Now, the problem here is how to teach a child.
  • fast_forward00:43:27 - Which internal concept, internal state a word is used to speak about.
  • fast_forward00:43:35 - Because you cannot see the internal state and you cannot point to an object and say, and this is joy.
  • fast_forward00:43:43 - So you can and there's a problem.
  • fast_forward00:43:48 - And the answer, which actually goes back to the language philosopher Wittgenstein,
  • fast_forward00:43:53 - he says He says, well, I can teach the meaning of these words because the child
  • fast_forward00:44:02 - has a natural tendency of expressing these internal states in behavior, in actions.
  • fast_forward00:44:08 - So if I want to teach a child what joy means, there's a very simple pathway.
  • fast_forward00:44:18 - Way i wait until the the child shows
  • fast_forward00:44:22 - joy behavior expresses joy in in its behavior and then i say well you're you're
  • fast_forward00:44:28 - joyful today all right yeah and and this way they i can establish a correlation
  • fast_forward00:44:33 - uh between between the word use and the and and the limbic activation which by assumption would.
  • fast_forward00:44:43 - Produce this behavior.
  • fast_forward00:44:45 - However, the link only works because there's an expression in behavior.
  • fast_forward00:44:50 - There's a manifestation of the internal state in behavior, in actions,
  • fast_forward00:44:56 - and therefore the motor system.
  • fast_forward00:44:59 - Comes in here and we actually showed actually
  • fast_forward00:45:02 - my doctorate student rachel mostly in cambridge she
  • fast_forward00:45:05 - she showed that for abstract emotion words the
  • fast_forward00:45:08 - most highly abstract emotion word there is motor systems
  • fast_forward00:45:12 - activation and interestingly it's those parts of the
  • fast_forward00:45:15 - motor system that are usually used to
  • fast_forward00:45:18 - express emotions namely the face and the hands so
  • fast_forward00:45:21 - it's the upper part of the body so this is
  • fast_forward00:45:24 - really amazing right because you're you're basically saying of course there's
  • fast_forward00:45:27 - a little little caveat because limbic system might also
  • fast_forward00:45:30 - involve then subcortical structures no that's no
  • fast_forward00:45:33 - caveat at all okay that's part of the game no but
  • fast_forward00:45:36 - earlier you wanted to stick to the neocortex as your mixer your mixing system
  • fast_forward00:45:41 - yes so now now you start to bring in so maybe it's a bit broader than only absolutely
  • fast_forward00:45:45 - absolutely so so this this semantic model is not restricted to the cortex It
  • fast_forward00:45:51 - has these limbic tails, if you like.
  • fast_forward00:45:55 - It's good that we agree on that. So now, what you say is very profound,
  • fast_forward00:46:01 - but on the other hand, it might also appear very sort of, in some sense, obvious.
  • fast_forward00:46:06 - Because you could say what your observation is,
  • fast_forward00:46:10 - so over a large number of experiments using different techniques in humans that,
  • fast_forward00:46:15 - okay, meaning in the end is expressed in a broad response in the brain that
  • fast_forward00:46:21 - is expressing, let's say, the experience of the subject with whatever the evoking event is.
  • fast_forward00:46:28 - And if it's an emotional component, emotion areas of the brain will come in.
  • fast_forward00:46:33 - If it's an action component, action areas will come in, et cetera.
  • fast_forward00:46:36 - But then you could say okay but.
  • fast_forward00:46:40 - Isn't that in some sense obvious because I'm just sort of mirroring the statistics
  • fast_forward00:46:45 - of the world in which I exist and this is then again reflected and then it seems so unspecific,
  • fast_forward00:46:53 - so that means it's not so the problem there is maybe also that you rely for
  • fast_forward00:47:00 - instance a lot on fMRI, fMRI is a fairly slow signal.
  • fast_forward00:47:04 - So, this might make it difficult to actually understand what really the core
  • fast_forward00:47:09 - is of that meaning network.
  • fast_forward00:47:11 - There might be a beginning and an end to this meaning network.
  • fast_forward00:47:15 - Maybe what you're looking at are like really the last few ripples in the semantic
  • fast_forward00:47:19 - system that indeed have sort of pervaded the system to really its periphery
  • fast_forward00:47:26 - in meaning space in this case.
  • fast_forward00:47:28 - And you have not really looked at the core of that meaning system.
  • fast_forward00:47:32 - Yeah. What would be the criterion for core and distant relationship?
  • fast_forward00:47:39 - Well, let me try to rephrase your question. Of course, if I hear a word,
  • fast_forward00:47:44 - I would first understand its meaning, and then I would think,
  • fast_forward00:47:47 - and then the word may remind me of something else.
  • fast_forward00:47:51 - I may have a second-order process of being reminded of an event.
  • fast_forward00:47:55 - I hear about freeing, and then I'm reminded of the news last night about some
  • fast_forward00:48:04 - report from a certain country,
  • fast_forward00:48:09 - and then I think about that and reprocess that and think about the freeing action
  • fast_forward00:48:14 - some rebels have performed there.
  • fast_forward00:48:19 - And of course this takes time after a while I may think about very distantly
  • fast_forward00:48:24 - related issues is this what you mean for instance yeah and so here I,
  • fast_forward00:48:30 - We have, of course, a handle to address this experimentally.
  • fast_forward00:48:36 - FMRI, as you said, isn't probably the best tool here because it's sluggish,
  • fast_forward00:48:40 - it's slow, it's like a snail.
  • fast_forward00:48:44 - And we cannot distinguish whether this activation, strictly speaking,
  • fast_forward00:48:48 - of the hand motor cortex when hearing the word grasp or to write,
  • fast_forward00:48:55 - whether this activation of the hand region in this case is actually the result
  • fast_forward00:49:00 - of understanding the word or relates to the understanding of the word or thinking
  • fast_forward00:49:05 - twice about it or imagining something distantly related to the word. We cannot.
  • fast_forward00:49:11 - No way. Because hopping from one association to the next in this experiment
  • fast_forward00:49:17 - may take only 200-400 milliseconds and this is not the temporal resolution of MRI for that.
  • fast_forward00:49:24 - We need imaging with exact temporal resolution, with millisecond precision.
  • fast_forward00:49:32 - EEG is good for that. MEG, magnetoencephalography, is ideal.
  • fast_forward00:49:37 - And we have done such experiments. And in a nutshell,
  • fast_forward00:49:41 - we see in MEG and EEG experiments that these motor systems activation,
  • fast_forward00:49:47 - for example, also the differential activation of visual areas in color and form-related
  • fast_forward00:49:57 - word processing come up very quickly within 200 milliseconds.
  • fast_forward00:50:03 - And this is as early as the earliest signs of semantic processes.
  • fast_forward00:50:08 - But can you say something about the sequence in which this is unfolding?
  • fast_forward00:50:12 - So I give you a word. It says, let's say, grasp.
  • fast_forward00:50:16 - Yes. So what will be now the sequence of events in the brain.
  • fast_forward00:50:22 - That show you that this semantic network is unfolding.
  • fast_forward00:50:27 - So we started with, let's say, you're reading it. So first we have some response
  • fast_forward00:50:31 - in the early visual system.
  • fast_forward00:50:33 - But what's next? Is it really unfolding in a very sequential way,
  • fast_forward00:50:37 - like we're slowly crawling up this hierarchy?
  • fast_forward00:50:39 - Or do you jump, let's say, to a more forward frontal area that could be,
  • fast_forward00:50:46 - let's say, more abstract, and from there you go back into these modality-specific
  • fast_forward00:50:49 - representations? What's the order in which this really occurs?
  • fast_forward00:51:20 - And from there, activation spreads to other regions, the posterior superior
  • fast_forward00:51:28 - temporal region and inferior frontal cortex, and as well to the motor system.
  • fast_forward00:51:34 - So, for example, with spoken words, we have measured with MEG a delay of peak
  • fast_forward00:51:41 - activations in superior temporal cortex of about 120 milliseconds after the
  • fast_forward00:51:48 - point of word recognition when the subject could first identify the upcoming word.
  • fast_forward00:51:54 - Of course, if you have a word like crocodile, you cannot be sure whether it
  • fast_forward00:52:00 - will be crocodile or crocus.
  • fast_forward00:52:01 - But at one stage, you are sure it will be crocodile and not something else.
  • fast_forward00:52:06 - And at that point, it takes another 120 or so milliseconds until we see this
  • fast_forward00:52:13 - activation peak already in superior temporal cortex.
  • fast_forward00:52:17 - And it's another 20 milliseconds more until the inferior frontal cortex.
  • fast_forward00:52:23 - Cortex and the lower part of the motor strip comes in.
  • fast_forward00:52:27 - And for leg words, the delays are a little bit larger.
  • fast_forward00:52:32 - Activation has to travel to the top of the motor strip.
  • fast_forward00:52:36 - So there we measured 170 milliseconds. So there's a slight difference in the time delays.
  • fast_forward00:52:44 - It's rather quick. 130 milliseconds in the back, 150 in the front,
  • fast_forward00:52:50 - and 170 at the top of the motor strip.
  • fast_forward00:52:52 - And interestingly, already these inferior frontal and superior central activations
  • fast_forward00:53:00 - at 140, 50, 60, 70 milliseconds,
  • fast_forward00:53:04 - they show a modulation which can be related, correlated with meaning aspects of the words. Okay.
  • fast_forward00:53:13 - But now, do you see this unfolding process as being tightly regulated by some,
  • fast_forward00:53:18 - let's say, a hub somewhere that's connected to all these structures that is
  • fast_forward00:53:22 - regulating how they unfold?
  • fast_forward00:53:23 - This has been a proposal, an important proposal by Karamazov, for example.
  • fast_forward00:53:29 - So the idea is, well, let me just rephrase this colloquially,
  • fast_forward00:53:34 - that this motor system activation is more or less for fun,
  • fast_forward00:53:37 - but that there's somewhere a semantic hub or a symbolic semantic system which pulls all strings.
  • fast_forward00:53:45 - Strings, and at one stage activation may overflow to the motor system,
  • fast_forward00:53:50 - there may be a little bit of a contribution here and there, but the semantic
  • fast_forward00:53:54 - system, the real semantic processing should be done elsewhere.
  • fast_forward00:53:59 - Now, we don't see this. We don't see this generally, that activation always
  • fast_forward00:54:04 - goes to temporal pole, for example, or to a different region before it goes to motor systems.
  • fast_forward00:54:12 - And we see for abstract sentence and abstract concept processing,
  • fast_forward00:54:20 - we see some activation which is outside the motor system, outside sensory systems,
  • fast_forward00:54:25 - for example, in dorsolateral prefrontal cortex and anterior.
  • fast_forward00:54:29 - Parts of the inferior frontal cortex.
  • fast_forward00:54:33 - But these activations, they usually come up at latencies that are very similar
  • fast_forward00:54:40 - to those where also modification of the motor strip activation reflecting semantic aspects,
  • fast_forward00:54:50 - referential semantic aspects, have been observed.
  • fast_forward00:54:54 - A recent experiment by my colleague Veronique Boulanger,
  • fast_forward00:54:58 - who did a postdoc in Cambridge a few years ago,
  • fast_forward00:55:01 - clarified this issue very nicely by looking at idiomatic expressions such as
  • fast_forward00:55:08 - grasp the idea or kick the habit. bit.
  • fast_forward00:55:20 - So these are kick and grasp sentences, hand sentences and leg sentences and
  • fast_forward00:55:26 - they activated to a degree differentially the motor strip and,
  • fast_forward00:55:31 - And this differential motor strip activation happened around 150 to 200 milliseconds,
  • fast_forward00:55:37 - the earliest significant differences were there.
  • fast_forward00:55:41 - Now, we also had literal sentences in this experiment and compared those with idiomatic ones.
  • fast_forward00:55:49 - And this general idiomaticity difference, that was.
  • fast_forward00:55:54 - Especially nicely reflected in dorsolateral prefrontal cortex,
  • fast_forward00:55:58 - not in the motor system, interestingly and also
  • fast_forward00:56:01 - in anterior temporal cortex so and and
  • fast_forward00:56:05 - and now one could of course have a race when is
  • fast_forward00:56:08 - this general symbolic difference abstract versus
  • fast_forward00:56:11 - versus versus concrete meaning idiomatic versus versus literal when is this
  • fast_forward00:56:19 - when does this first come up which would allow for a conclusion on any general
  • fast_forward00:56:24 - symbolic system activation if if one wanted to phrase it this way.
  • fast_forward00:56:30 - And when would this so-called embodied activation of the motor system,
  • fast_forward00:56:34 - differential between arm and leg sentences, even though they are abstract, come up?
  • fast_forward00:56:42 - And we use this trigger point here, not of course the action-related words,
  • fast_forward00:56:47 - but those words that disambiguated the meaning between idiomatic or concrete
  • fast_forward00:56:56 - literal sentence meaning,
  • fast_forward00:56:58 - so the point of when they catch the ball or catch the apple,
  • fast_forward00:57:06 - this noun at the end would then distinguish between the literal and the idiomatic meaning,
  • fast_forward00:57:13 - or grasp the idea deer
  • fast_forward00:57:16 - or grass to
  • fast_forward00:57:19 - cup yeah exactly and when when
  • fast_forward00:57:22 - those come up and out and and it
  • fast_forward00:57:25 - was just about 150 milliseconds after this word was flashed on screen that we
  • fast_forward00:57:30 - saw both the this prefrontal anterior temporal idiomaticity effect and also
  • fast_forward00:57:37 - the the hand leg dissociation in the motor strip So the embodied,
  • fast_forward00:57:43 - there's parity between the embodied and this potentially more general idiomaticity effects.
  • fast_forward00:57:55 - It's not so that one can easily be said to drive the object.
  • fast_forward00:58:00 - Exactly. So this is an important consequence of this, right?
  • fast_forward00:58:02 - So in your mind, this is not really orchestrated by some hidden module somewhere.
  • fast_forward00:58:08 - No, there isn't a semantic homunculus. Right, exactly. Exactly.
  • fast_forward00:58:10 - This is really just evolving out of the network dynamics without a central orchestration.
  • fast_forward00:58:16 - Yes. The orchestration is actually, as we would say, is done by the neuronal
  • fast_forward00:58:23 - units that can be widely distributed.
  • fast_forward00:58:26 - Exactly. But there's not a central conductor who says now you and now you. Yes, exactly.
  • fast_forward00:58:31 - So this is good, right? So here we have this dynamical response,
  • fast_forward00:58:36 - which will vary dependent on the references to the outside world.
  • fast_forward00:58:40 - But on the other hand, there's this issue of specificity because your test case
  • fast_forward00:58:44 - is very much action words, if you want, where you can show specificity to some extent.
  • fast_forward00:58:50 - If it's, let's say, a hand word, you would see more, let's say,
  • fast_forward00:58:53 - hand-related responses in the sematosensory system.
  • fast_forward00:58:59 - While it's kicking a foot action, you would see more foot-related responses.
  • fast_forward00:59:03 - But you were also fair enough this morning to show results from some of your
  • fast_forward00:59:09 - colleagues or competitors, if you want, who showed that actually these responses
  • fast_forward00:59:13 - might not be that specific,
  • fast_forward00:59:15 - that you also might see responses in this motor area to completely nonsensical
  • fast_forward00:59:19 - words or non-action words.
  • fast_forward00:59:23 - So this might be a threat to your theory because they say, look,
  • fast_forward00:59:27 - these things might be responding all the time.
  • fast_forward00:59:30 - Not quite. Okay. Because motor semantics or action semantics is not the only
  • fast_forward00:59:35 - thing that drives the motor system.
  • fast_forward00:59:38 - So we have shown in a little series of experiments that also phonological features
  • fast_forward00:59:44 - of words and nonsense sounds drive the motor system.
  • fast_forward00:59:52 - And we even hear it to natural sound.
  • fast_forward00:59:55 - We even see this motor system's activation to natural sounds.
  • fast_forward00:59:59 - For example, take this out.
  • fast_forward01:00:02 - My colleague, Olive Hoke, in Cambridge, he did an experiment where he used some neutral sounds,
  • fast_forward01:00:13 - like a metronome, and then occasionally there was an unexpected sound.
  • fast_forward01:00:26 - These were clicks, either just clicks that have no body relationship.
  • fast_forward01:00:32 - Or it was sounds produced by the tongue, sounds produced by the finger.
  • fast_forward01:00:37 - And interestingly, he found for these entirely non-linguistic sounds,
  • fast_forward01:00:44 - activation of different parts of the motor strip.
  • fast_forward01:00:48 - Very early, very similar to our word, evoked activation, but it's not specific
  • fast_forward01:00:53 - to the meaning of words in no way.
  • fast_forward01:00:56 - And when looking now at brain activation elicited by language sounds that are
  • fast_forward01:01:06 - produced with a tongue or with the lips, like p,
  • fast_forward01:01:10 - t, t, p,
  • fast_forward01:01:12 - p, k, they are either sounds produced by the tongue, sounds produced by the lips.
  • fast_forward01:01:21 - When people hear this,
  • fast_forward01:01:24 - we see a trace of activation of the motor strip that reflects the body part
  • fast_forward01:01:29 - relationship of these sounds, namely the articulator that plays the biggest
  • fast_forward01:01:35 - role in the articulation,
  • fast_forward01:01:36 - either the tongue or the lips.
  • fast_forward01:01:39 - So there, even for nonsense words, for example, those starting with a P or a
  • fast_forward01:01:45 - T, there's good reason to see activation in the, at least in the inferior part of the motor strip.
  • fast_forward01:01:52 - So this is not damaging at all to the semantics proposal.
  • fast_forward01:01:57 - Of course, it calls for very careful experimental manipulation, because if I compare….
  • fast_forward01:02:06 - Words that start with t with hand semantics to words that start with with p and uh with with,
  • fast_forward01:02:14 - face relationship semantic face relationship then there may be a severe confound
  • fast_forward01:02:20 - and experiment results maybe may look different much different from if if this
  • fast_forward01:02:26 - phonological features were were matched exactly so so this is but another interesting
  • fast_forward01:02:31 - consequence i think of your work is that you can say,
  • fast_forward01:02:33 - look, maybe in terms of our linguistic conventions.
  • fast_forward01:02:37 - The word might not have such a motor component, but maybe for your brain it does.
  • fast_forward01:02:42 - And we just don't really understand yet how that's wired into a broader semantic system.
  • fast_forward01:02:47 - So in that sense, maybe it's not so much a challenge to your theory.
  • fast_forward01:02:51 - It also opens a new window on how we can rethink semantics of maybe even non-action words, right?
  • fast_forward01:02:58 - There will be a whole set of non-action words that is able to trigger activation in the motor system.
  • fast_forward01:03:03 - So possibly in the semantic networks the brain is building up,
  • fast_forward01:03:07 - there are actually action components that we have overlooked in our linguistic analysis.
  • fast_forward01:03:11 - Yes. Yeah. And the most obvious example here are these abstract emotion words,
  • fast_forward01:03:18 - where nobody would have imagined an activation of the motor system.
  • fast_forward01:03:23 - There's semantically, Basically, if one asks about action features or object
  • fast_forward01:03:29 - relationship referential features, there's no nothing there.
  • fast_forward01:03:33 - But one wouldn't say this is what one uses these words to speak about actions.
  • fast_forward01:03:38 - One would say it would speak about internal states, but nevertheless,
  • fast_forward01:03:43 - semantic theory of a different kind would stipulate, postulate that the meaning
  • fast_forward01:03:51 - of these words can only be learned by way of having actions.
  • fast_forward01:03:59 - That are manifestations, criteria for the presence of these internal states.
  • fast_forward01:04:05 - Right. But then… So this is something that's a new contribution.
  • fast_forward01:04:09 - Exactly. And a contribution that actually speaks to an issue which is of great
  • fast_forward01:04:13 - relevance for the theory of language and semantics and philosophy of language.
  • fast_forward01:04:19 - Absolutely. No, I completely get the point. And I don't think anyone has really
  • fast_forward01:04:24 - followed it up yet within the linguistic community, or am I wrong?
  • fast_forward01:04:29 - Have the linguists responded yet to this? Well, it would be a bit quick because
  • fast_forward01:04:33 - the paper has just been published a few months ago.
  • fast_forward01:04:36 - Yeah, but in some sense it's already a consequence of earlier work that,
  • fast_forward01:04:39 - for instance, that you see responses in motor areas to non-action words is already
  • fast_forward01:04:45 - raising that question. Yes. Right?
  • fast_forward01:04:47 - So that question is now on the table for some time. Yes.
  • fast_forward01:04:52 - So that should be followed up. But now, what you have done, your theory,
  • fast_forward01:04:57 - which in some sense goes back to Donald Happ and it might even go back to Thorndike
  • fast_forward01:05:02 - who also would say the brain is, you know, a connection machine. Yes.
  • fast_forward01:05:07 - And in some sense, you postulate a very simple rule. You say,
  • fast_forward01:05:10 - look, you know, the brain just follows the statistics of its inputs.
  • fast_forward01:05:13 - And if these statistics linguistics coincide with
  • fast_forward01:05:16 - a word yeah then we have semantics yes
  • fast_forward01:05:20 - indeed and but of course when when saying this
  • fast_forward01:05:23 - in the in a cognitive theory context uh well there are some colleagues who look
  • fast_forward01:05:30 - at you as if the devil was confronted with holy water so to speak that's right
  • fast_forward01:05:37 - you are the devil in that analogy or the holy water i was about to phrase it the other way around.
  • fast_forward01:05:44 - Well, you can see it from whatever perspective you approach it.
  • fast_forward01:05:51 - So it was a sacrilegious, it was an entire no-go to speak about correlations
  • fast_forward01:05:58 - in such abstract domains as semantics.
  • fast_forward01:06:01 - And some people are really very much opposed to that.
  • fast_forward01:06:09 - And this has a certain relationship to statements that had been made also in
  • fast_forward01:06:15 - the behaviorist tradition.
  • fast_forward01:06:18 - Of course, relating the meaning of a word to actions and stimuli and responses
  • fast_forward01:06:26 - that had been, such ideas had been.
  • fast_forward01:06:29 - That sounds like Skinner's theory of language. Exactly, yeah.
  • fast_forward01:06:32 - There is a relationship. Now, if you read Skinner, this is much too primitive to work at any level.
  • fast_forward01:06:39 - And, of course, it was exactly the behaviorists who denied any internal states.
  • fast_forward01:06:45 - So, theorizing about cell assemblies in the cortex that represent cognitive
  • fast_forward01:06:50 - representations would be an entire no-go.
  • fast_forward01:06:54 - Well, except when you would talk with more cognitive behaviorists like Tolman and Hull.
  • fast_forward01:07:00 - So, I think Skinner and Watson were a bit more at one extreme end of the scale.
  • fast_forward01:07:04 - But, yes, you seem to be alluding to these kinds of concepts.
  • fast_forward01:07:08 - And of course, that's asking for trouble when you are confronted with domains
  • fast_forward01:07:12 - that are still more dominated by this cognitivist perspective.
  • fast_forward01:07:15 - Yeah, but I think leaving aside all these to a degree religious ideas,
  • fast_forward01:07:23 - we should be clear that we want to understand cognitive mechanisms and we want
  • fast_forward01:07:30 - to spell out their basis in terms of neural circuits.
  • fast_forward01:07:34 - We are not satisfied and many, many, I think the majority I can say is not satisfied
  • fast_forward01:07:41 - any longer with box and arrow diagrams.
  • fast_forward01:07:44 - We want to become more concrete and using certain standard perceptron architectures
  • fast_forward01:07:53 - is still a little bit distant from cortical anatomy and function as well.
  • fast_forward01:08:02 - So we are trying to use models that are more inspired by cortical anatomy.
  • fast_forward01:08:08 - What would be the minimal ingredients of the model?
  • fast_forward01:08:12 - What are the minimum components the model should have to build up these kinds
  • fast_forward01:08:17 - of semantic representations?
  • fast_forward01:08:19 - I guess we need sort of neural units. They need to have a firing state.
  • fast_forward01:08:24 - They need to have connections. These connections must be able to change dependent on activity levels.
  • fast_forward01:08:30 - But what are then the minimum agreed ingredients of this model to replicate
  • fast_forward01:08:35 - your results? Well, there would need to be a relationship between the model
  • fast_forward01:08:39 - and its parts and the brain.
  • fast_forward01:08:42 - And that's not so easy because there are many areas that are of relevance.
  • fast_forward01:08:48 - And there's the back part of the language cortex, the pericelial regions,
  • fast_forward01:08:54 - the front part, there's motor-premotor, prefrontal cortex, there's auditory
  • fast_forward01:09:01 - cortex, auditory belt, auditory parabelt.
  • fast_forward01:09:04 - Well, then visual cortices come in.
  • fast_forward01:09:09 - Other motor fields for controlling hand and leg actions are important.
  • fast_forward01:09:15 - So it's quite a range of different areas and their connectivity.
  • fast_forward01:09:19 - So it's quite a little bit of work. But you could argue, well,
  • fast_forward01:09:22 - I don't know, maybe that's easier than it sounds because certainly if we want
  • fast_forward01:09:26 - to restrict ourselves to the neocortex, you do know that the different areas
  • fast_forward01:09:31 - you point to in terms of the local circuits are relatively similar.
  • fast_forward01:09:36 - Yes. So in terms of the computational unit of your model, it might actually
  • fast_forward01:09:42 - be fairly straightforward.
  • fast_forward01:09:44 - Or not? No. Okay, why not? Because the whole architecture influences the nature
  • fast_forward01:09:51 - of the computational unit.
  • fast_forward01:09:53 - It just happens that if we have correlated activation in such a network on the
  • fast_forward01:10:00 - motor end and on the auditory end of the network,
  • fast_forward01:10:04 - that then waves of activation spread through the various areas of the,
  • fast_forward01:10:11 - let's call them areas instead of layers, areas of the network.
  • fast_forward01:10:15 - And therefore, populations of neurons strengthen their connections all over the network.
  • fast_forward01:10:22 - And what we finally end up is our cell assemblies,
  • fast_forward01:10:26 - neuronal assemblies, neuronal representations that link between motor and auditory
  • fast_forward01:10:32 - visual input-output patterns and have a center actually in those areas that
  • fast_forward01:10:41 - link motor systems to auditory and visual systems.
  • fast_forward01:10:45 - Like, in fact, the prefrontal cortex and some higher association temporal parietal region.
  • fast_forward01:10:55 - And those are particularly important for holding together these distributed units of processing.
  • fast_forward01:11:03 - So, we get what Hebb postulated, namely these cell assemblies.
  • fast_forward01:11:09 - And in Breitenberg's hands, actually, the proposal was made for the first time
  • fast_forward01:11:15 - that these networks could actually bridge between cortical areas.
  • fast_forward01:11:19 - And this is also what we find in the simulations.
  • fast_forward01:11:23 - And what we think is the best explanation for this extremely rapid spreading,
  • fast_forward01:11:28 - which relates to aspects of the meaning of words and construction.
  • fast_forward01:11:34 - Right, so in some sense that would mean, okay, there's a, let's say there's
  • fast_forward01:11:38 - an input to this network, to the brain.
  • fast_forward01:11:41 - The input triggers different areas. They start to propagate their activities
  • fast_forward01:11:46 - through the rest of the network.
  • fast_forward01:11:47 - These activation patterns might coincide in different places.
  • fast_forward01:11:51 - I pick that up in my connections, and now I have, let's say,
  • fast_forward01:11:55 - additional memory-related responses to these semantic events.
  • fast_forward01:12:00 - This would be roughly the right interpretation.
  • fast_forward01:12:02 - Yes, yes. But wouldn't that imply then you would have hubs, no?
  • fast_forward01:12:05 - Then you are actually involving hubs in your network. Well, what we end up with
  • fast_forward01:12:10 - after this learning exercise are functional units.
  • fast_forward01:12:16 - It's actually quite in their behavior, not analog, but discrete processing units.
  • fast_forward01:12:22 - These cell assemblies, they are strongly connected networks,
  • fast_forward01:12:25 - and partial activation of a significant part of the network would lead to an
  • fast_forward01:12:32 - explosion-like process,
  • fast_forward01:12:34 - activation spreading through the network.
  • fast_forward01:12:38 - And so the cell assembly becomes active.
  • fast_forward01:12:41 - And there's actually a danger of having activation spreading everywhere.
  • fast_forward01:12:46 - And we need inhibition, we need to have regulation, regulation mechanisms in
  • fast_forward01:12:52 - order to keep down the general level of activation and make sure that with one
  • fast_forward01:12:56 - activation of a word or conceptual or semantic representation,
  • fast_forward01:13:01 - we do not activate 10 more within an instance.
  • fast_forward01:13:04 - So we have strong inhibition, and this inhibition could be titrated so that
  • fast_forward01:13:15 - only one cell assembly becomes active at a time.
  • fast_forward01:13:18 - Of course, if it's not properly adjusted, you get… Yeah, it will explode, basically.
  • fast_forward01:13:24 - Explode schizophrenia or epileptic networks.
  • fast_forward01:13:30 - But no, there's another…,
  • fast_forward01:13:33 - prediction here i'm not sure if that's the one you would like to see but,
  • fast_forward01:13:37 - um if signals travel in the brain you will
  • fast_forward01:13:40 - have a transduction latency of the signal with the distance it has to travel
  • fast_forward01:13:44 - yes and this the largest the semantic net the potential semantic network is
  • fast_forward01:13:50 - rather let's say roughly could be anything in that in the neocortex so we forget
  • fast_forward01:13:54 - limbic system but that means i'm going to link together areas at varying distances,
  • fast_forward01:14:00 - So, these intersection points, right, where the activity will overlap will vary
  • fast_forward01:14:06 - with the distances between these areas, right?
  • fast_forward01:14:08 - Because I have rhythmic activity, there's a transduction latency.
  • fast_forward01:14:11 - So, the points where these activation waves will overlap, so learning can occur
  • fast_forward01:14:15 - following your rule, will vary with, let's say, the initial points of activity.
  • fast_forward01:14:21 - So you get regular tessellation of these hubs or however, these intermediate
  • fast_forward01:14:27 - points dependent on the initial spatial configuration that you trigger.
  • fast_forward01:14:32 - Is it also something you see in your fMRI data?
  • fast_forward01:14:39 - That with different word types we see, well.
  • fast_forward01:14:46 - Maybe it's not clear. No, I wasn't fully with you.
  • fast_forward01:14:50 - Are you aiming at a potential imaging difference between one of these areas
  • fast_forward01:15:01 - that hold together the cell assembly and the peripheral?
  • fast_forward01:15:06 - Let's do auditory action words that are spoken, and I'm reading action words.
  • fast_forward01:15:12 - Yeah. So now the distance from temporal cortex or my auditory responses and
  • fast_forward01:15:18 - the motor cortex, that distance is different from the distance between my occipital
  • fast_forward01:15:24 - cortex, my visual response, and the motor cortex.
  • fast_forward01:15:26 - So now both these areas will become activated in some way, right?
  • fast_forward01:15:31 - So now their activity starts to propagate through the broader network,
  • fast_forward01:15:35 - but the intersection points of this activation will be at different distances
  • fast_forward01:15:39 - in between these areas because their distance is different.
  • fast_forward01:15:43 - And if your network model is correct and would predict,
  • fast_forward01:15:47 - I see activity islands building up somewhere between temporal and motor when
  • fast_forward01:15:53 - it's auditory and more, let's say, more towards occipital when it's visual motor. This is correct.
  • fast_forward01:16:02 - And we see, for example, in the latency that with visual word presentation,
  • fast_forward01:16:07 - the delay of this motor systems activation is a little bit delay, a little bit larger.
  • fast_forward01:16:13 - So it's 200 to 220, 40 milliseconds.
  • fast_forward01:16:17 - Now with auditory, as I said before, it's 150 to 160, 70 milliseconds.
  • fast_forward01:16:21 - Second so it's earlier with the auditory with the earlier presentation as as far as we can draw um uh.
  • fast_forward01:16:31 - This this comparison because of course there are
  • fast_forward01:16:34 - basic differences between auditory presentation which
  • fast_forward01:16:37 - is piecemeal crop or dial i i
  • fast_forward01:16:40 - get a phoneme by phoneme uh now with with crocodile being presented flashed
  • fast_forward01:16:46 - on the screen i have all the information in one shot right it could also be
  • fast_forward01:16:50 - that this this heavy shot of information uh requires more time and therefore
  • fast_forward01:16:56 - the delay the delay is larger.
  • fast_forward01:16:58 - So we cannot uniquely attribute these delay differences to cortical distance.
  • fast_forward01:17:05 - There's possibly an influence of the nature of the experiment.
  • fast_forward01:17:10 - But as far as this differential activation dynamics are concerned,
  • fast_forward01:17:19 - there are indeed differences in the delays and also in the pathway of the cortical spreading. Okay.
  • fast_forward01:17:25 - But is it also expressed in, let's say, the activation of areas in between these…
  • fast_forward01:17:32 - Yes, especially inferior frontal cortex, Broca's region, seems to play a role. Okay.
  • fast_forward01:17:38 - But another consequence of the model that I was wondering about is that your
  • fast_forward01:17:43 - model is highly sensitive to statistics, right?
  • fast_forward01:17:45 - So the more components my semantics has, so the more combinatorial it gets,
  • fast_forward01:17:54 - the more input states I have to sample.
  • fast_forward01:17:57 - So the prediction of that would be that in order to build up a response to a
  • fast_forward01:18:02 - word that has more components, I need more exposure.
  • fast_forward01:18:06 - I need to see it more often because I'm frequency dependent.
  • fast_forward01:18:10 - So learning will depend on the frequency with which things occur.
  • fast_forward01:18:14 - Occur so do you do you observe that
  • fast_forward01:18:16 - as well in your fmri data let's say let's say words action
  • fast_forward01:18:20 - words that are less frequent will trigger different responses weaker
  • fast_forward01:18:22 - responses than action words that are more frequent well
  • fast_forward01:18:26 - we have a we have a degree of these are not very very strong effects but there
  • fast_forward01:18:31 - there's a there's a degree of uh of of variability of this for example these
  • fast_forward01:18:36 - motor systems and and and middle temporal cortex activations varying with word frequencies as Okay,
  • fast_forward01:18:44 - but would you take that as a prediction of your model?
  • fast_forward01:18:50 - Well, as you said, or at least implied, not only the frequency as such should
  • fast_forward01:18:56 - be relevant, but also the complexity of the semantic relationship.
  • fast_forward01:19:02 - So, if very variable semantic relationships exist of one given word, then of course the,
  • fast_forward01:19:14 - additional learning steps may just lead to a weakening of the connection.
  • fast_forward01:19:18 - So if I have, for example, a word that is used to speak about,
  • fast_forward01:19:26 - about a wide range of objects and then water animals and then a word with the
  • fast_forward01:19:38 - same frequency that is specifically used,
  • fast_forward01:19:42 - to speak about one particular type of animal like frog.
  • fast_forward01:19:47 - And then, of course, the semantic links may be very strong in this.
  • fast_forward01:19:53 - Or unicorn. Unicorn would be not a difficult case, right? Yeah, indeed.
  • fast_forward01:19:59 - Yeah. Yeah, unicorn would be especially difficult. Exactly.
  • fast_forward01:20:02 - One would need to build it from different parts. Right. Horns and horses.
  • fast_forward01:20:10 - But another amazing implication of your work is that you have pursuits.
  • fast_forward01:20:15 - Like we talked earlier about this controversy, if you want, with more cognitivist approaches.
  • fast_forward01:20:20 - Yes. Get out of here. This is too easy. It will never work like this.
  • fast_forward01:20:24 - This is not possible. And so on.
  • fast_forward01:20:28 - But you have taken actually, I think, a very courageous step by saying,
  • fast_forward01:20:32 - okay, look, if this is all correct, I must be able to actually treat aphasia with this effectively.
  • fast_forward01:20:37 - Yeah. So what was the concept there? How did you make that switch? What's the idea?
  • fast_forward01:20:43 - Well, if you want the true and honest answer, it was the other way around.
  • fast_forward01:20:49 - Okay. I started, it was aphasia therapy that started it all.
  • fast_forward01:20:55 - Okay, interesting. I did my PhD in linguistics then on aphasia therapy,
  • fast_forward01:21:03 - being convinced that if a theory has something to say, it must have a practical implication.
  • fast_forward01:21:10 - This wasn't very popular in the last century, I should say.
  • fast_forward01:21:16 - It still isn't that much, right? And I did this in a linguistics department,
  • fast_forward01:21:21 - and the linguist said, well, is this really linguistics what this guy is doing?
  • fast_forward01:21:27 - But I was kind of imprinted a little bit by my grandfather who suffered from
  • fast_forward01:21:35 - a severe aphasia, and this always was a problem, and nobody could help him essentially.
  • fast_forward01:21:42 - And so after studying linguistics, after learning about language therapy and
  • fast_forward01:21:50 - also studying biology and learning
  • fast_forward01:21:52 - about brain mechanisms, I thought that specializing in this direction,
  • fast_forward01:21:56 - learning about brain and language relationships in order to produce something
  • fast_forward01:22:01 - sensible for helping these patients would be an important thing to do.
  • fast_forward01:22:08 - And then as a PhD student, I made a few proposals based on semantic theory,
  • fast_forward01:22:17 - language philosophy, and some...
  • fast_forward01:22:22 - Knowledge about neurobiological mechanisms.
  • fast_forward01:22:26 - And this resulted then after, this was then not very successful.
  • fast_forward01:22:34 - At the start, 1991, the first paper was published and was not received at all.
  • fast_forward01:22:42 - And about 10 years later, I met with a famous U.S.
  • fast_forward01:22:48 - Neuroscientist, Edward Taub, who had developed constrained-use movement therapy over dinner.
  • fast_forward01:22:54 - He said, well, wouldn't you have an idea how we could project this into the
  • fast_forward01:22:59 - language domain and link up with some behaviorally relevant language training
  • fast_forward01:23:05 - similar to the motor therapy?
  • fast_forward01:23:08 - And then I reactivated my previous.
  • fast_forward01:23:14 - Interest and experience also.
  • fast_forward01:23:18 - The idea of the therapy was to embed language in action context,
  • fast_forward01:23:24 - actually in interactions, communicative interactions and this was very handy
  • fast_forward01:23:32 - also for use in a in a mass practice context and we then ran a a randomized controlled trial,
  • fast_forward01:23:42 - and it was extremely successful.
  • fast_forward01:23:44 - That paper on aphasia therapy was really very well received,
  • fast_forward01:23:52 - and we are continuing with this.
  • fast_forward01:23:55 - And I hope that we will, at one stage, be successful to secure some money to do some important work.
  • fast_forward01:24:03 - Work to optimize it further, to link it to computational work and to have computer
  • fast_forward01:24:14 - versions of the therapy so that the intensity couldn't go up even further. Exactly.
  • fast_forward01:24:19 - But in your case, you could interpret your approach also as if you want an inverted
  • fast_forward01:24:26 - mirror approach, right?
  • fast_forward01:24:27 - Because in mirroring, the mirror mechanisms, it's very often interpreted like,
  • fast_forward01:24:31 - okay, it's the sensory state perception, visual perception of action that drives
  • fast_forward01:24:38 - the motor execution pathway.
  • fast_forward01:24:40 - So we go from the sensory state, the perceptual state, to the motor state.
  • fast_forward01:24:44 - But in your case, you turn it around and say, no, no, wait, if I can tickle
  • fast_forward01:24:47 - the motor state by having the subject gesture,
  • fast_forward01:24:51 - I can actually now exactly follow that pathway backwards and tickle all the
  • fast_forward01:24:58 - areas that are sort of upstream from it,
  • fast_forward01:25:01 - that were under normal conditions talking to this motor system.
  • fast_forward01:25:05 - Yes. Is this a correct interpretation?
  • fast_forward01:25:07 - Yes. Okay. Okay, so then how effective is this as a therapeutic tool?
  • fast_forward01:25:14 - The idea is that the language system and the motor system or the action system
  • fast_forward01:25:19 - or the general action system of the brain, they are heavily linked to each other, interwoven. Yeah.
  • fast_forward01:25:25 - And that if one fails, if one suffers from a lesion due to the strong connection
  • fast_forward01:25:32 - to the other, this other system, this partially still intact system,
  • fast_forward01:25:38 - could help restoring neural patterns of activation.
  • fast_forward01:25:42 - And so that's the theory behind it. So lesion network could get help from a strong associate.
  • fast_forward01:25:49 - The language system could get help from the action system. The theory and the
  • fast_forward01:25:54 - data, they are pretty favorable.
  • fast_forward01:25:57 - So we did randomize controlled trials. And it seemed in one study that this
  • fast_forward01:26:01 - approach is more efficient than the therapy patients get.
  • fast_forward01:26:08 - That's in chronic patients or acute patients? Chronic. We have been restricting
  • fast_forward01:26:13 - ourselves to chronic patients for methodological reasons because in acute patients,
  • fast_forward01:26:20 - there's such an unpredictable amount of spontaneous recovery that any conclusions
  • fast_forward01:26:29 - of a randomized controlled trial is always in danger of receiving criticism.
  • fast_forward01:26:38 - Because there could be just randomly, you could just by chance choose good recoverers
  • fast_forward01:26:46 - and have bad ones in the other group.
  • fast_forward01:26:50 - So you're never safe. But if you go to stages where spontaneous recovery is
  • fast_forward01:26:55 - not so likely and minimal normally, then this possibility can be excluded.
  • fast_forward01:27:02 - We chose patients who were several years after the stroke in the average.
  • fast_forward01:27:07 - Which I think it was six or so.
  • fast_forward01:27:09 - Okay. Yes, and so quite a long time.
  • fast_forward01:27:12 - But in some sense, it might have started with a discussion on constraint-induced therapy. Yes.
  • fast_forward01:27:19 - But what are the constraints you are imposing?
  • fast_forward01:27:23 - Because essentially you are mobilizing the system, asking people to do more
  • fast_forward01:27:27 - than they would usually do.
  • fast_forward01:27:28 - Well, the term constraint-induced has produced a lot of misunderstanding,
  • fast_forward01:27:34 - understanding and there's this association of the, of the, uh,
  • fast_forward01:27:40 - of torture, doing something bad to already poor patients.
  • fast_forward01:27:46 - So we try to promote the term intensive language action therapy,
  • fast_forward01:27:54 - which I think might be more appropriate.
  • fast_forward01:27:57 - Constraints are nevertheless important. We wouldn't speak about constraints
  • fast_forward01:28:02 - because of the negative connotation, but we would, to formulate it more positively,
  • fast_forward01:28:10 - I would say we focus the patients on their communicative needs.
  • fast_forward01:28:16 - In these patients, as in Ed Taub's motor therapy patients, there's a natural
  • fast_forward01:28:24 - tendency of avoiding whatever is complicated, whatever is usually not successful.
  • fast_forward01:28:29 - So as a patient, if I realize that certain complex words do not work normally, I just stop trying.
  • fast_forward01:28:38 - Yes. And if I cannot do complex sentences, I try to avoid that and just use the words I can do.
  • fast_forward01:28:47 - I may end up with telegraphic style.
  • fast_forward01:28:50 - And this is what we want to avoid.
  • fast_forward01:28:54 - We want to really force them to try their best.
  • fast_forward01:28:57 - And in order to that, we constrain or we focus the therapy,
  • fast_forward01:29:01 - and we introduce rules in the interaction games so that the communication task
  • fast_forward01:29:12 - is difficult enough to challenge them.
  • fast_forward01:29:15 - Right, exactly. To challenge, if possible, each patient.
  • fast_forward01:29:18 - Right. So we titrate the requirements for each patient in order to have optimal practice.
  • fast_forward01:29:25 - Right. And I think that's very important, even though constraint,
  • fast_forward01:29:30 - The word constraint has produced many misunderstandings, to say the least. Yeah, exactly.
  • fast_forward01:29:38 - But I think it's really marvelous that you have made that step towards the clinic,
  • fast_forward01:29:43 - which is, as you know, extremely challenging.
  • fast_forward01:29:47 - Certainly, if you talk about rehabilitation of stroke, a lot of people stay
  • fast_forward01:29:51 - away from these kinds of, because usually things fail in that domain.
  • fast_forward01:29:54 - It's just really hard. And it's really amazing that you pushed that so far and
  • fast_forward01:29:58 - also had such an impact. And that, of course, gives you, I think… It was because
  • fast_forward01:30:02 - I started this stupid enterprise before I knew how complicated it is. Right, exactly.
  • fast_forward01:30:09 - And therefore, I was in that business already, and I was actually positively
  • fast_forward01:30:14 - surprised. Right, exactly. That's really impressive.
  • fast_forward01:30:16 - But also, this gives you ammunition, I think, in this debate with the more,
  • fast_forward01:30:21 - let's say, theory-inclined cockativists, because at least you can show you have real impact.
  • fast_forward01:30:26 - So now, to finish up my two questions to which you already have been exposed, so you're prepared.
  • fast_forward01:30:37 - So working on this issue of language, semantics, and the brain,
  • fast_forward01:30:41 - and also aphasia treatment, so what's the law of Friedemann that we should follow
  • fast_forward01:30:47 - in trying to understand the brain?
  • fast_forward01:30:48 - Well, I wouldn't search for any Freedmanian law, but if anything,
  • fast_forward01:30:59 - I would speak of a Breitenbergian approach to mechanizing cognition.
  • fast_forward01:31:05 - To look out for, if possible,
  • fast_forward01:31:13 - the simplest possible neural instantiations, neural mechanistic circuits that could underlie...
  • fast_forward01:31:28 - Complex cognitive abilities, such as meaning processing. And we actually want
  • fast_forward01:31:33 - to draw circuits that do the job,
  • fast_forward01:31:35 - and we want to simulate the buildup and the activation of these circuits in.
  • fast_forward01:31:44 - Realistic computer models that mimic the brain.
  • fast_forward01:31:47 - We want to use these neurocomputational models to predict cortical activation
  • fast_forward01:31:56 - patterns, predict and then explain cortical activation patterns.
  • fast_forward01:32:00 - And if it doesn't come out in the experiments like this, we want to adjust these
  • fast_forward01:32:06 - neural models in order to improve them and to get a better idea about the mechanisms.
  • fast_forward01:32:13 - I think that's the pathway to make progress to better understand cognition.
  • fast_forward01:32:20 - I think we have seen these box and arrow diagrams now for 30, 40,
  • fast_forward01:32:27 - 50 years, and I think it's really time now to go in a mechanistic direction
  • fast_forward01:32:36 - where we theorize not in terms of abstract mechanisms,
  • fast_forward01:32:41 - but in terms of concrete neuronal mechanisms that solve abstract problems. Exactly.
  • fast_forward01:32:48 - Very good. No, I like that. So then five years from now, I'm going to go visit you there in Cambridge.
  • fast_forward01:32:56 - Not in Cambridge, in Berlin now. Oh, you're in Berlin now. Sorry, yes. My mistake. Yeah.
  • fast_forward01:33:02 - And I'm going to confront you with a hypothesis you're going to generate today.
  • fast_forward01:33:06 - So what's the one hypothesis you're most passionate about today that you believe
  • fast_forward01:33:12 - will be confirmed five years from now?
  • fast_forward01:33:20 - Um the hypothesis well they
  • fast_forward01:33:25 - are hypotheses related to the language therapy enterprise and
  • fast_forward01:33:28 - of course optimizing this further and showing that that that the motor system
  • fast_forward01:33:33 - indeed activates more after this therapy and reflects actually the motor system
  • fast_forward01:33:40 - activations or strength stronger links between motor and language systems actually
  • fast_forward01:33:44 - are the result of successful language action therapy.
  • fast_forward01:33:48 - That would be a big hope to make progress in this direction.
  • fast_forward01:33:55 - Improving the therapy further has very high priority for me.
  • fast_forward01:34:01 - Now, if your question aims more at the theoretical level,
  • fast_forward01:34:07 - there is a very important question which I would like to learn something about,
  • fast_forward01:34:10 - which links to your question about the difference between conceptual and semantic system.
  • fast_forward01:34:19 - And I said that for the time being, we could consider semantics a subset of
  • fast_forward01:34:32 - concepts or conceptual space.
  • fast_forward01:34:37 - Now, this is not entirely correct.
  • fast_forward01:34:40 - Because there's an idea that.
  • fast_forward01:34:43 - Linking concepts to language also helps
  • fast_forward01:34:47 - conceptual processing and the
  • fast_forward01:34:50 - and the the idea is that the different
  • fast_forward01:34:53 - differentiation of concepts discrimination of very similar concepts is only
  • fast_forward01:34:59 - possible if we link it to a to a symbol system such as language the reason being
  • fast_forward01:35:04 - that That if you do neurocomputational models and try to build representations
  • fast_forward01:35:11 - for very similar things,
  • fast_forward01:35:13 - for a crocodile and for an alligator.
  • fast_forward01:35:16 - We may end up with cell assemblies, with neural representations that overlap very much.
  • fast_forward01:35:24 - Well, of course, we can use neurocomputational tricks such as Kohonen networks
  • fast_forward01:35:28 - where we force separate non-overlapping representations.
  • fast_forward01:35:33 - If you don't do this, and we just map the cell assemblies and the overlap of
  • fast_forward01:35:40 - these networks mirror the semantic similarity, then we are in a hopeless situation.
  • fast_forward01:35:45 - We have 95% overlap of semantic features, and then the two glue together.
  • fast_forward01:35:50 - We cannot separate them. Hopeless.
  • fast_forward01:35:52 - What we can do is use tricks in
  • fast_forward01:35:57 - order to separate the two, to make these representations more dissimilar.
  • fast_forward01:36:02 - If we manage to link alligator with one word, namely the word alligator,
  • fast_forward01:36:09 - and the other concept, the crocodile concept, with an entirely different word, namely crocodile,
  • fast_forward01:36:15 - then this union of the conceptual and the word representation,
  • fast_forward01:36:21 - So, the entire semantic network, they would now,
  • fast_forward01:36:26 - if the conceptual overlap was 95%, then the overlap of the linguistic conceptual
  • fast_forward01:36:35 - semantic network would only be below 50%,
  • fast_forward01:36:40 - 47.4%.
  • fast_forward01:36:42 - So, it would give you some form of, let's say, a top-down decorrelation. Right.
  • fast_forward01:36:46 - Indeed, of course there's a problem how to link them up with each other if they
  • fast_forward01:36:52 - are already overlapping so much in the first place,
  • fast_forward01:36:54 - but assuming that this can be solved with special tricks and inhibitory mechanisms.
  • fast_forward01:37:03 - So the specific prediction would be that semantics as language-related representations
  • fast_forward01:37:11 - is not only built on top of a conceptual space,
  • fast_forward01:37:15 - it in turn recurrently helps you to organize and boost the conceptual space.
  • fast_forward01:37:19 - Exactly. So the idea is language helps conceptual discrimination.
  • fast_forward01:37:24 - Okay. And in five years, 10 years, 15 years… No, it was five.
  • fast_forward01:37:28 - I would like to have clear evidence for that. Great. Well, Friedman Pulvermoor,
  • fast_forward01:37:34 - thank you very much for this conversation.
  • fast_forward01:37:35 - Thanks so much. This was fantastic. And I hope we can do something in the future
  • fast_forward01:37:40 - on language action therapy.
  • fast_forward01:37:43 - That would be fantastic. That would be great, yes. Wonderful. Absolutely.
  • fast_forward01:37:47 - Thanks so much for the interview. Thank you. It was a great pleasure.
  • fast_forward01:37:51 - The CSN Podcast was produced by the Convergent Science Network of Biometrics
  • fast_forward01:37:57 - and Biohybrid Systems, A project funded by the European Sevens Research Framework Programme.
  • fast_forward01:38:06 - Music.

Be the first to leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

Convergent PozitivLinia Logo

Exploring the convergence of neuroscience, robotics, and AI through conversations with leading researchers since 2010.

A project of the Convergent Science Network Foundation.

© CSN Podcasts. Developed by IMCreativeWEBC

0%

Login to enjoy full advantages

Please login or subscribe to continue.

Go Premium!

Enjoy the full advantage of the premium access.

Stop following

Unfollow Cancel

Cancel subscription

Are you sure you want to cancel your subscription? You will lose your Premium access and stored playlists.

Go back Confirm cancellation