cover giovanni pezzulo

Giovanni Pezzulo on predictive brain and embodied cognition

  • cover play_arrow

    PLAY EPISODE


cover giovanni pezzulo
Season 2012
Season 2012
Description arrow_drop_down

Description

Watch an expert rock climber study a wall they have never seen before, and you are watching the motor system think. Giovanni Pezzulo explains how the predictive brain reuses sensorimotor knowledge for problem solving, imagery, and understanding other minds.

Subscribe for more from the Convergent Science Network podcast series.

Pezzulo distinguishes two kinds of prediction that are often conflated in the literature. Implicit prediction, as in classical conditioning, attaches value labels to stimuli without maintaining an internal model of the predictive relationship. Explicit prediction builds structured forward models of environmental regularities that can be run offline for planning, decision-making, and mental simulation. The predictive brain hypothesis proposes that the brain systematically incorporates environmental structure into such models and uses them to drive perception, attention, and action selection proactively rather than reactively.

The interview centers on embodied problem solving, illustrated by competitive rock climbers who study an unfamiliar wall before ascending. Expert climbers visibly rehearse motor sequences, moving their arms to simulate grasps and reaches, using their bodies as external scaffolds for cognition. This is not mere motor programming: the climber must assemble partial skills in novel combinations, evaluate reachability constraints, and explore a vast space of possible routes, all guided by proprioceptive knowledge that only expertise provides. Memory experiments confirm that expert climbers remember difficult routes significantly better than novices, but only when the routes are actually climbable, demonstrating that motor expertise structures perception and memory rather than providing a generic cognitive advantage.

Pezzulo builds from individual action to social cognition through a series of escalating steps. If your motor system generates predictions about your own actions, it can also predict the actions of others by running the same forward models with different parameters. This simulation-based understanding of others bootstraps joint action planning, coordination, and eventually the ability to influence another person’s beliefs and intentions. Clinical evidence supports this continuum: patients with bilateral parietal lesions cannot inhibit imagined actions from becoming overt movements, and utilization behavior patients automatically grasp objects they see, revealing the tight coupling between internal simulation and motor execution that normally remains covert.

Tagged as:

About the author call_made

CSN Podcasts

Both the triumphs of humanity and its most evil deeds have resulted from collaboration. In a time where humanity is required to aspire to the former and minimize the latter, the question arises of how collaboration arises and why it fails. Surprisingly, this phenomenon, so central to who we are, is not well understood. Hence, a collaborative effort is required to understand collaboration in its full biological, psychological, sociological, cultural, and economic complexity and to translate this understanding into operational impact. This series of podcasts is one step toward achieving these complementary goals. The Collaboration Podcast presents interviews with people who are central orchestrators of collaboration in various domains including business, government, science, art, health, sustainability, and the military. The discussions were conducted by Prof. Dr. Paul F.M.J. Verschure and members of the Program Advisory Committee of the Ernst Strungmann Forum on Collaboration (https://www.esforum.de/forums/ESF32_Collaboration.html) during 2021 and had the goal to sketch a map of opportunities, challenges, and obstacles in human collaboration. The forum took place in May 2022, and now we would like to share this series of interviews with a broader audience. The full report of the Forum will be published in 2023 by MIT Press. The podcast was produced by the Convergent Science Network (https://www.convergentsciencenetwork.org/). Context: The stability of social systems depends critically on realizing sustainable methods of “collaboration,” yet how and by which means collaboration is achieved is not clearly understood; neither are the conditions or processes that lead to its breakdown or failure. Collaboration can be understood as cooperation between agents toward mutually constructed goals. Part of the reason for our lack of understanding is that the phenomenon of collaboration is, by nature, a highly multidisciplinary problem, and effective research into its complexities has been difficult to achieve across the broad range of scientific and technical disciplines involved. The need for a fundamental understanding of collaboration, however, has become increasingly important. Not only does humankind demand answers as it attempts to address critical challenges at multiple scales (e.g., climate change, migration, enhanced automation, social and economic inequality), but ever-increasing technological and economic means of interconnecting people and societies are disrupting long-established, familiar patterns of how we interact. Radical technological changes that are ongoing have the potential to reshape collaboration in ways that are currently hard to predict or influence (e.g., by altering configurations in interaction, information creation, and modes of communication). On one hand, such changes could disrupt hitherto stable forms of collaboration by affecting critical communication channels and traditional roles, as can be observed in the rapidly changing patterns in governance, commerce, and social interaction. Conversely, technology could lead to the emergence of novel, successful forms of collaboration that deviate from traditional “hierarchical” architectures. Evidence of this can be seen in areas as diverse as highly automated manufacturing plants, the open science movement, collaborative software repositories, user-centered services, and the sharing of economy-based modes of organization. Without a fundamental understanding of the mechanisms, processes, and boundary conditions of collaboration, it is not possible to evaluate or predict which of these possible scenarios are sustainable or even plausible. The Forum “How Collaboration Arises and Why it Fails” (May 8–13, 2022, Location: Frankfurt am Main, Germany) Chairs: Andreas Roepstorff and Paul Verschure Program Advisory Committee: Jenna Bednar, Julia R. Lupp, Bhavani R. Rao , Andreas Roepstorff, Ferdinand von Siemens, and Paul Verschure

More posts

Timestamp

  • fast_forward00:00:03 - This is the Convergent Science Network podcast. Leading researchers in the domain
  • fast_forward00:00:10 - of neuroscience, brain theory and technology are interviewed by Paul Verschure and Tony Prescott.
  • fast_forward00:00:20 - This is Paul Verschure with the Convergent Science Network podcast.
  • fast_forward00:00:24 - And in this episode, I'm speaking with Giovanni Pizzulo, who is a speaker at
  • fast_forward00:00:29 - our summer school, Barcelona Cognition Brain and Technology summer school.
  • fast_forward00:00:34 - And Giovanni, you started out with investigating this whole issue of the predictive brain.
  • fast_forward00:00:42 - Yes. So what does it mean for your predictive brain exactly?
  • fast_forward00:00:46 - Well, in a sense, the predictive brain is just a big idea.
  • fast_forward00:00:51 - Well, the idea is that the brain is not a passive organ.
  • fast_forward00:00:54 - So rather than just sitting down and waiting for the next stimulus,
  • fast_forward00:00:58 - in a sense, it really always tries to anticipate what comes next,
  • fast_forward00:01:02 - to set up some internal goals, and so to have some big internal processing.
  • fast_forward00:01:07 - So rather than being stimulus-based, it's really trying to anticipate the necessities
  • fast_forward00:01:12 - or maybe the opportunities for action.
  • fast_forward00:01:14 - So that's really the big principle that drives all the brain processing because
  • fast_forward00:01:19 - now, because the brain has some predictions, some goals, it can really organize
  • fast_forward00:01:23 - its sensory processing, its attention processes,
  • fast_forward00:01:26 - and its motor preparation processes for quickly responding and doing some adaptive actions.
  • fast_forward00:01:33 - So to do that, it has to be predictive and proactive. Mm-hmm.
  • fast_forward00:01:37 - But then where does that really start? What are the good examples of this kind of prediction?
  • fast_forward00:01:43 - Well, in the literature, there are many, many partially disconnected literature on prediction.
  • fast_forward00:01:50 - So predictions in sensory processing, predictions in the motor domain,
  • fast_forward00:01:54 - predictions also in higher cognition, in that, for instance,
  • fast_forward00:01:56 - I can try to anticipate what your next question will be.
  • fast_forward00:02:00 - So all these literatures can now proceed a little bit disconnected.
  • fast_forward00:02:05 - But for instance, in the sensory domain, you see a lot of predictive stuff going
  • fast_forward00:02:11 - on, such as the anticipation of the next stimulus.
  • fast_forward00:02:15 - In the motor control domain, you really have to get rid of what are the consequences
  • fast_forward00:02:21 - of your action for many reasons.
  • fast_forward00:02:23 - One big reason in the motor domain is that sensory perceptions are very ambiguous,
  • fast_forward00:02:28 - so by also predicting what comes next, you better estimate the state of the world.
  • fast_forward00:02:33 - Another big reason is for decision-making. So if you can really anticipate what
  • fast_forward00:02:38 - the effects of your action are,
  • fast_forward00:02:39 - then you can also select them on different courses of actions beforehand.
  • fast_forward00:02:43 - Beforehand, such as, for instance, if I anticipate that by going left,
  • fast_forward00:02:47 - I will get some big reward, and by going right, I will get some big punishment,
  • fast_forward00:02:53 - then that's the decision.
  • fast_forward00:02:55 - But now, in some sense, let's say also the physiological study of learning that
  • fast_forward00:03:00 - started with Pavlov, one of
  • fast_forward00:03:03 - the first observations of Pavlov was that the brain is predicting, right?
  • fast_forward00:03:07 - So that's also a key feature of classical conditioning. So in some sense,
  • fast_forward00:03:11 - the concept of prediction is with us for a long time.
  • fast_forward00:03:15 - And as some speakers have said, like sometimes certain things get so old that
  • fast_forward00:03:19 - they sound new again, right?
  • fast_forward00:03:20 - So what is actually really new in this current movement of the brain as a predictor?
  • fast_forward00:03:28 - Well, actually, I completely agree that also in classical conditioning,
  • fast_forward00:03:33 - you have this predicting dynamics going on.
  • fast_forward00:03:35 - But I think that we should try to really distinguish at least two kinds of predictions.
  • fast_forward00:03:43 - It's not a sharp distinction, but it's useful to do probably.
  • fast_forward00:03:45 - So one is an implicit kind of prediction in that, for instance,
  • fast_forward00:03:50 - in the conditioning studies.
  • fast_forward00:03:52 - So a stimulus, which is typically highly predictive of another stimulus,
  • fast_forward00:03:57 - which is in turn highly predictive of reward.
  • fast_forward00:04:00 - So the first stimulus, which is predictive of another stimulus,
  • fast_forward00:04:03 - it becomes itself good to achieve.
  • fast_forward00:04:06 - So in that case the brain becomes implicitly
  • fast_forward00:04:10 - predictive in that it stores the relationships between
  • fast_forward00:04:13 - the first and the second stimulus but to do
  • fast_forward00:04:16 - that you don't really need to to maintain into
  • fast_forward00:04:19 - your memory or to an internal representation of the
  • fast_forward00:04:22 - predictive relation between the two you just attach some good label to the to
  • fast_forward00:04:29 - the first stimulus whereas there is a second kind of prediction which is a more
  • fast_forward00:04:34 - explicit prediction in which you keep into your brain some model of the environment.
  • fast_forward00:04:40 - So the point is that in the first case you really have some implicit mechanisms
  • fast_forward00:04:46 - that link good or bad to sequences of things, whereas in the second mechanism
  • fast_forward00:04:51 - you also maintain a model of these sequences of things.
  • fast_forward00:04:54 - For instance in the motor domain you can, for instance, let's say you have to grasp a moving ball.
  • fast_forward00:05:02 - There are two different ways to do that. So one way is to look at the moving
  • fast_forward00:05:07 - ball and then jump to some future state.
  • fast_forward00:05:11 - To some future state that the ball will reach at some point.
  • fast_forward00:05:16 - This can be done in two ways. The first way is just learning the contingencies automatically.
  • fast_forward00:05:21 - So I know that if I see the ball moving, then I have to go to some position,
  • fast_forward00:05:29 - which is not the current position, but the future position.
  • fast_forward00:05:32 - Without even anticipating that position explicitly, you can do that automatically.
  • fast_forward00:05:36 - The second way to do that is really doing what is called a forward model,
  • fast_forward00:05:44 - so a predictive model that really tells you where the ball will be in the future.
  • fast_forward00:05:48 - Use this forward model online, so this predictive mechanism online,
  • fast_forward00:05:52 - and then move the hand to the predicted position.
  • fast_forward00:05:58 - So to recapitulate this idea, there are implicit and explicit mechanisms,
  • fast_forward00:06:03 - and the novelty of or the predictive brain hypothesis, if you want,
  • fast_forward00:06:07 - is the hypothesis that the brain systematically incorporates,
  • fast_forward00:06:12 - regularities of the external environment in a structured way so as to form strong
  • fast_forward00:06:17 - models, statistical models, for instance, of the regularities of the external environment.
  • fast_forward00:06:22 - And then it systematically uses these models to plan what to do next and also
  • fast_forward00:06:28 - to drive perception, to drive attention, and so on.
  • fast_forward00:06:31 - So, whereas in the first formulation of prediction,
  • fast_forward00:06:35 - it was more an implicit prediction, in the predictive brain hypothesis,
  • fast_forward00:06:39 - at least in some of its formulation, it's much more about building models for
  • fast_forward00:06:44 - the world, on top of which you can run explicit predictions. Right, so then….
  • fast_forward00:06:49 - If you would look at, let's say, the theoretical literature on the predictive
  • fast_forward00:06:53 - brain, what in your mind are right now the outstanding examples of this approach?
  • fast_forward00:06:59 - Well, there are many of them.
  • fast_forward00:07:03 - So probably in a very famous paper by Rao and Ballard, it was already in the
  • fast_forward00:07:11 - 99, if I remember well, it was about this predictive coding idea,
  • fast_forward00:07:16 - predictive coding idea in perception.
  • fast_forward00:07:18 - So the idea is that the brain continuously generates prediction about the stimulus,
  • fast_forward00:07:22 - and it uses these predictions in a top-down manner, whereas it uses prediction errors,
  • fast_forward00:07:30 - so the difference between the prediction and the sensory evidence,
  • fast_forward00:07:33 - as a revision mechanism.
  • fast_forward00:07:36 - So that was a milestone in a sense.
  • fast_forward00:07:39 - But, well, there are many other papers now. There is a very big framework put
  • fast_forward00:07:44 - forward by Carl Freestone, which is called the Free Energy Framework or the
  • fast_forward00:07:48 - Active Inference Framework.
  • fast_forward00:07:50 - In that sense, Freestone really tries to look at the whole brain,
  • fast_forward00:07:53 - not just the perceptual system, as a big prediction machine.
  • fast_forward00:07:59 - There are many other frameworks. One is put forward by Moshibar,
  • fast_forward00:08:04 - again, in the perceptual domain,
  • fast_forward00:08:05 - whereas in the motor domain, also the leading view, one of the most authoritative
  • fast_forward00:08:09 - view put forward by people such as Danny Wolpert or by Shadmer or many other
  • fast_forward00:08:15 - people, is that in reality,
  • fast_forward00:08:19 - what you really predict is the sensory consequences of your actions.
  • fast_forward00:08:24 - So that framework is more tied to motor predictions.
  • fast_forward00:08:27 - Other people, such as Mark Sherrow or others, they have tried to expand this
  • fast_forward00:08:34 - framework from the prediction of the sensory consequences of actions to more
  • fast_forward00:08:39 - complicated forms of cognition,
  • fast_forward00:08:40 - such as reusing this prediction for understanding of the action of others,
  • fast_forward00:08:45 - such as reusing this prediction for imagery,
  • fast_forward00:08:48 - and so thinking at more abstract situations.
  • fast_forward00:08:50 - So, I would say that there are many fields in which prediction have been studied,
  • fast_forward00:08:56 - and also some converging frameworks.
  • fast_forward00:09:01 - Right. So now, if we talk about your own experiments in the role of prediction in problem solving,
  • fast_forward00:09:09 - you showed some examples of what you called embodied problem solving. Yes.
  • fast_forward00:09:13 - Right? So how does that, which aspects of the predictive brain are these experiments exercising?
  • fast_forward00:09:23 - Well, it's not so simple to explain the videos that I've shown,
  • fast_forward00:09:28 - but just to recap, it's just a video of a climber.
  • fast_forward00:09:32 - And maybe you know, but climbers prior to a competition, they have some time
  • fast_forward00:09:38 - to look at the climbing wall that they will then climb during the competition.
  • fast_forward00:09:43 - The nice point is that they see this climbing wall for the first time and there
  • fast_forward00:09:47 - are many in these climbing walls,
  • fast_forward00:09:48 - there are many climbing holes arranged all over or through the wall and the
  • fast_forward00:09:52 - climber has one minute or a few minutes for figuring out how to better climb
  • fast_forward00:09:58 - this wall and if you look at the video or if you look at the climbing competition you see people,
  • fast_forward00:10:04 - People, climbers really moving the arms such as to really anticipate the next moves.
  • fast_forward00:10:11 - They see moving the arms in one direction, then the other direction.
  • fast_forward00:10:13 - That's really a form of problem solving because they see the climbing wall for the first time.
  • fast_forward00:10:18 - They have to figure out a good plan for action.
  • fast_forward00:10:21 - I call it problem solving, not only planning, because it's really complex.
  • fast_forward00:10:25 - There are many constraints. It depends on where you go.
  • fast_forward00:10:27 - Then you can reach or cannot reach the other hold. So you can also find in your
  • fast_forward00:10:32 - mind many, many solutions, compare them.
  • fast_forward00:10:35 - And in this specific kind of setups, the hypothesis is that it is really the
  • fast_forward00:10:43 - motor system that is governing this process.
  • fast_forward00:10:48 - So expert climbers are really able to anticipate the force they have to put into this motor act.
  • fast_forward00:10:55 - They can really anticipate a lot of proprioceptive information.
  • fast_forward00:10:58 - They can really figure out if a hold is in their reach or out of reach,
  • fast_forward00:11:04 - if this hold is too little to be really grasped, if it is too far away for them,
  • fast_forward00:11:08 - then they have to take another route.
  • fast_forward00:11:10 - So that's, I call it embodied problem solving for many reasons.
  • fast_forward00:11:14 - So one reason is that you really use knowledge of your body and of your motor
  • fast_forward00:11:18 - system to figure out how to better solve this problem.
  • fast_forward00:11:21 - Problem and another reason is that it's also
  • fast_forward00:11:24 - overtly embodied in that you really use your
  • fast_forward00:11:27 - body your body movements as a scaffold as a help a helping tool for solving
  • fast_forward00:11:31 - the problem that's a bit tricky to explain without the video but in a sense
  • fast_forward00:11:35 - by simply moving your yourself you don't need to to keep everything into your
  • fast_forward00:11:40 - memory you use your body as part of the problem solving process.
  • fast_forward00:11:45 - Okay, but now you could also see it as a form of, let's say, motor programming.
  • fast_forward00:11:50 - That you say, look, I have to execute a movement sequence very rapidly. Okay.
  • fast_forward00:11:56 - Um, and I'm just going to now rehearse this movement sequence that I can sort
  • fast_forward00:12:00 - of automatically trigger one set of behavioral motion motor patterns after the
  • fast_forward00:12:06 - other, and it can climb up the wall more rapidly.
  • fast_forward00:12:09 - Yeah, that's, that's part of the, that's part of the story. So the point is
  • fast_forward00:12:13 - forming this motor program.
  • fast_forward00:12:15 - Uh, the, the reason why I call it an embodied problem solving is that the solution is not so trivial.
  • fast_forward00:12:21 - So the point is that you have to try to form this motor program by assembling
  • fast_forward00:12:26 - partial skills that you used in the past in novel ways.
  • fast_forward00:12:34 - So there is a lot of flexibility in this new assemblage of motor or small motor programs.
  • fast_forward00:12:40 - Yeah, but the problem solving would suggest that there is also,
  • fast_forward00:12:44 - let's say, a goal state in the world that you want to achieve in order that
  • fast_forward00:12:47 - you also have to manipulate aspects of that world.
  • fast_forward00:12:50 - Yeah. Well, in this case, it's essentially making sure your body goes through
  • fast_forward00:12:55 - a certain sequence of motions.
  • fast_forward00:12:57 - So how would your view on embodied problem solving then generalize to problem solving in general?
  • fast_forward00:13:04 - Because I would assume that you want to identify some more, let's say,
  • fast_forward00:13:07 - generic aspects of problem solving as opposed to specialized aspects of problem solving.
  • fast_forward00:13:13 - Yeah, okay. So there are two parts of this story. So one part is that even in
  • fast_forward00:13:17 - that case, even in this climbing example, you have a goal. So the goal is reaching the top.
  • fast_forward00:13:24 - And then it is not simply running through a trajectory, but also finding out
  • fast_forward00:13:30 - which is the good trajectory.
  • fast_forward00:13:32 - So it's not so different from the Tower of London or other typical problem-solving setup.
  • fast_forward00:13:38 - So you have a goal state, and you have many, many sequences of moves that you
  • fast_forward00:13:43 - can do, and you explore this space of possibilities.
  • fast_forward00:13:47 - But the trick is that you explore it in intelligent ways, so you don't simply try out all of them.
  • fast_forward00:13:53 - You really use your expertise to find out how to better explore this.
  • fast_forward00:13:58 - The second point is that you really use knowledge incorporated into your body,
  • fast_forward00:14:02 - your motor programs, to figure out what are the constraints of the problem.
  • fast_forward00:14:06 - The climbing problem has many constraints because, as I said before,
  • fast_forward00:14:10 - if you are too much to the left of the climbing wall, you cannot reach any more the holds on the right.
  • fast_forward00:14:16 - Or vice versa, if you jump too high, you cannot probably make your body stiff
  • fast_forward00:14:24 - enough to really hold the climbing hold.
  • fast_forward00:14:27 - There are many constraints that you really discover while solving the problem
  • fast_forward00:14:33 - by reenacting this motor knowledge. That's my point.
  • fast_forward00:14:36 - And the second part of the answer was about how much this generalizes to more
  • fast_forward00:14:41 - complex and abstract problem solving.
  • fast_forward00:14:43 - So, well, first of all, this is just a nice video that I showed just for illustrating
  • fast_forward00:14:47 - the possibility of solving problems by reusing motor knowledge or sensory motor knowledge.
  • fast_forward00:14:53 - It could also be affective knowledge. So knowledge incorporated into the same
  • fast_forward00:14:57 - models that you use for acting in the external world.
  • fast_forward00:15:00 - So I think that as a general strategy, that's the way we should look at higher cognition.
  • fast_forward00:15:06 - The challenge is looking at how you solve higher cognitive skills,
  • fast_forward00:15:10 - such as problem solving in abstract domains, by reusing these strategies that
  • fast_forward00:15:16 - you acquired so as to efficiently deal with the external environment.
  • fast_forward00:15:21 - So when you have these sensory-motor strategies, then probably our earlier ancient
  • fast_forward00:15:28 - evolutionary ancestors, they
  • fast_forward00:15:32 - had only these simple strategies to deal with their current situations.
  • fast_forward00:15:37 - Whereas higher cognitive skills is mostly about very complex abstract situations,
  • fast_forward00:15:43 - non-perceptually available events, distal goals.
  • fast_forward00:15:45 - But the challenge is seeing how simple strategies that we used in the past to
  • fast_forward00:15:52 - survive in the environment can be sophisticated and reused in these more complex cognitive domains.
  • fast_forward00:15:58 - So that's a challenge. My example was just to illustrate the possibility that
  • fast_forward00:16:03 - you can really solve problems by reusing systematically knowledge incorporated
  • fast_forward00:16:07 - into your internal models in an intelligent way.
  • fast_forward00:16:10 - And that allows also for a lot of flexibility. possibility is not for planning
  • fast_forward00:16:14 - the next action, but also for solving very complex problems.
  • fast_forward00:16:18 - That's a bit the wish, right? That's the wish. Yes. Then the question is,
  • fast_forward00:16:22 - how far are you in realizing that wish?
  • fast_forward00:16:26 - So first in the climber case, do you see a big difference between expert and novice climbers?
  • fast_forward00:16:33 - Yeah, absolutely. Yes. So part of the story, which is also one of the reasons
  • fast_forward00:16:38 - why I'm interested in it, is that But in a sense, it is the expertise that not
  • fast_forward00:16:44 - only modulates the climbing ability,
  • fast_forward00:16:46 - but also the ability to think about problems.
  • fast_forward00:16:49 - That's supporting for the embodied cognition view, for the view that it's really
  • fast_forward00:16:54 - knowledge incorporated into your motor system that helps you not only in executing the actions,
  • fast_forward00:16:59 - but also for instance in imagining the actions, in reusing the repertoire for solving problems.
  • fast_forward00:17:06 - At the same time, we have evidence that knowledge incorporated in your motor
  • fast_forward00:17:11 - system also helps you understand what other people are doing.
  • fast_forward00:17:16 - Right. So that's the idea in a sense.
  • fast_forward00:17:19 - Yeah, that's in the step into the social cognition component.
  • fast_forward00:17:23 - But in some sense what you're saying, look, the standard view would be,
  • fast_forward00:17:26 - let's say we perceive the world, we have some sort of high fidelity in the end
  • fast_forward00:17:32 - interpretation of the world on the basis we should make decisions and perform action.
  • fast_forward00:17:35 - But now the consequence of what you're saying is say, look, I'm
  • fast_forward00:17:38 - actually acting in this world and the way I act in the
  • fast_forward00:17:41 - world is now modulating or directly filtering the
  • fast_forward00:17:45 - way I'm going to perceive this world absolutely so the processing
  • fast_forward00:17:48 - goes backwards in that sense so the expert
  • fast_forward00:17:51 - climbing is looking at this world very differently as a novice climber yes and
  • fast_forward00:17:56 - you have evidence for that uh well we yes we perform them uh well of course
  • fast_forward00:18:02 - we have a tiny evidence up to now but i think the this evidence points in the
  • fast_forward00:18:06 - right direction so one memory study that we performed,
  • fast_forward00:18:10 - comparing novice and expert climbers.
  • fast_forward00:18:13 - So we simply asked these novice and expert climbers to look at three climbing routes,
  • fast_forward00:18:21 - one simple, that both of them were able to execute, one difficult,
  • fast_forward00:18:25 - that only the expert group was able to do, and one impossible,
  • fast_forward00:18:29 - which was not climbable, actually.
  • fast_forward00:18:31 - It was not really a climbing route. It was just a random displacement of climbing holds.
  • fast_forward00:18:37 - And so the task was simply remembering the climbing holds in the right sequence.
  • fast_forward00:18:44 - What we found is that in the easy condition that both novice and expert were
  • fast_forward00:18:49 - able to climb, both were also able to remember quite well.
  • fast_forward00:18:55 - So there was no evidence for a better remembering for experts.
  • fast_forward00:19:01 - While for the difficult route, only the experts were really able to remember it well.
  • fast_forward00:19:07 - And this, in our opinion, this points to the fact that really the way experts
  • fast_forward00:19:12 - structure the perceptually also the situation can also help in this memory task.
  • fast_forward00:19:19 - And so we had this control condition in which for the impossible route,
  • fast_forward00:19:25 - in that route, we did not find any advantage for the experts because this ability,
  • fast_forward00:19:30 - this increased memory ability is really tied to the climbability of the wall.
  • fast_forward00:19:34 - It's not a generic ability to remember climbing holds without context.
  • fast_forward00:19:40 - But to respond completely to your question, I would say that this sensory motor approach,
  • fast_forward00:19:48 - so this sensory motor influence of thinking on cognition, to me,
  • fast_forward00:19:52 - at least, it acts on two timescales, at least two timescales.
  • fast_forward00:19:56 - One timescale is learning and development.
  • fast_forward00:19:59 - So the point is that because I have some sensory motor skills,
  • fast_forward00:20:04 - then I change the way I perceive the world.
  • fast_forward00:20:06 - So I structure my memory and my perceptual abilities in such a way that then
  • fast_forward00:20:12 - supports better cognitive abilities.
  • fast_forward00:20:14 - So that acts on a longer time scale of learning and development.
  • fast_forward00:20:18 - The second way the sensory motor system supports cognition is more in the online cognition.
  • fast_forward00:20:23 - Cognition, because I can reenact my motor problems right now,
  • fast_forward00:20:27 - my motor programs right now, so I can use them so as to support,
  • fast_forward00:20:32 - for instance, imagery or action perception.
  • fast_forward00:20:35 - So there are two sides of the same coin. So in a sense, this framework predicts
  • fast_forward00:20:40 - that if you increase your sensory motor abilities in one domain,
  • fast_forward00:20:44 - then you really shape your perception, your memory, and all your cognitive processing.
  • fast_forward00:20:48 - And then you are also better able to run imaginary experiments based on this
  • fast_forward00:20:56 - motor expertise that you can now reenact.
  • fast_forward00:20:59 - Right. But then would the expert climbers recognize the unclimbable wall more
  • fast_forward00:21:05 - rapidly than the novices, novice climbers?
  • fast_forward00:21:10 - Well, probably, well, if the task is exactly recognized whether or not it is
  • fast_forward00:21:15 - climbable or unclimbable, that's the task.
  • fast_forward00:21:18 - If this is the task, well, of course, this is a bit of a tricky task because
  • fast_forward00:21:22 - unclimbable could be, is not such a clear concept.
  • fast_forward00:21:27 - But anyway, yes, I would say that in that case, what will happen is just that
  • fast_forward00:21:32 - the climbers try to climb it mentally.
  • fast_forward00:21:35 - So they run this imaginary climbing simulation, and then because the expert
  • fast_forward00:21:40 - will anticipate some failure in climbing, because they have high confidence
  • fast_forward00:21:45 - in their internal models, they will say, no, no, no, this cannot be climbed.
  • fast_forward00:21:49 - Whereas for the novices, maybe they will try out to anticipate.
  • fast_forward00:21:54 - They will also fail, but without noticing the, well, without trusting too much their simulations.
  • fast_forward00:22:01 - So that will be my... Right.
  • fast_forward00:22:04 - But now, still a problem for this point of view is that you could say,
  • fast_forward00:22:10 - look, the climber is a highly specialized human being.
  • fast_forward00:22:14 - So it's not very surprising that they have certain cognitive capabilities because
  • fast_forward00:22:19 - they have been overtrained in dealing with certain kinds of environments like climbing walls.
  • fast_forward00:22:24 - So how would you see this sort of.
  • fast_forward00:22:28 - But you want to look at this as an example where
  • fast_forward00:22:31 - you say well it's actually very basic sensory motor patterns or it's really
  • fast_forward00:22:37 - action itself and the way we generate action from which the rest of cognition
  • fast_forward00:22:42 - is sort of bootstrapped or is depends upon it's constrained by so how is it
  • fast_forward00:22:49 - going to work exactly how do you see that work,
  • fast_forward00:22:52 - Well, I would say that now we don't have any perfect story yet,
  • fast_forward00:22:57 - but the point is really that action and the goals implied in the action and
  • fast_forward00:23:02 - the predictions implied in the action, they are really key to cognition and
  • fast_forward00:23:06 - to develop increasingly more complex cognitive skills.
  • fast_forward00:23:09 - So at the beginning, so that could be a simple story.
  • fast_forward00:23:13 - So at the beginning, the brain is simply organized around these goal selection
  • fast_forward00:23:17 - and specification and selection tasks.
  • fast_forward00:23:19 - So they have to integrate information from the external world,
  • fast_forward00:23:22 - from the memory, from affective states, to very, very quickly jump into a good action selection.
  • fast_forward00:23:29 - And to do that, as I told before, you're mostly relying on your internally generated
  • fast_forward00:23:35 - goal state, affective processes, attentional processes.
  • fast_forward00:23:40 - The stimulus helps you, but then it's the brain, which is autonomous, is doing the job.
  • fast_forward00:23:45 - But then as the sensory motor abilities of this primitive architecture develop
  • fast_forward00:23:50 - over time and increase your abilities, they increase their ability to control
  • fast_forward00:23:54 - the external world and to predict it.
  • fast_forward00:23:57 - So they start incorporating more and more knowledge, more and more also structure
  • fast_forward00:24:04 - from the external environment into their predictions, into their motor control actions.
  • fast_forward00:24:09 - So how are you going to validate this prediction?
  • fast_forward00:24:13 - Is it a pure experimental exercise or are there other ways to get this validated?
  • fast_forward00:24:17 - The framework, you mean? Well, the framework, that's much more a lifelong research
  • fast_forward00:24:24 - program, if you want, because we are trying to do many, many validations.
  • fast_forward00:24:29 - A few empirical data, of course, already exist.
  • fast_forward00:24:32 - If you think of the literature, how similar the brain networks for imagery and
  • fast_forward00:24:37 - for execution or for motor preparation, they are.
  • fast_forward00:24:41 - So you already see that some mental operations are really supported by the sensory
  • fast_forward00:24:46 - motor system in the brain. So that's some evidence.
  • fast_forward00:24:50 - There is evidence, well, there are also patient studies that say that in some
  • fast_forward00:24:55 - conditions, the imaginary action cannot be really stopped from being overtly executed.
  • fast_forward00:25:02 - So that's another part of the story because
  • fast_forward00:25:05 - in this story it is it is the internalization of
  • fast_forward00:25:08 - the predictive mechanism that really supports cognition so you
  • fast_forward00:25:11 - first predict predict things in the external world then you internalize this
  • fast_forward00:25:15 - capability of predicting and this eventually lets you to rehearsing entire sequences
  • fast_forward00:25:20 - of action without executing them but then a prediction of this framework is
  • fast_forward00:25:25 - that if you really do this covert mental imagery if you want or discover covered predictive magnets,
  • fast_forward00:25:31 - and you cannot really separate these internal processes from the overt execution,
  • fast_forward00:25:38 - then this implies that what you imagine, you immediately do.
  • fast_forward00:25:42 - And there is evidence that this unfortunately happens in patients with bilateral parietal lesions.
  • fast_forward00:25:49 - And other evidence of this close connection between what you think and what you do is,
  • fast_forward00:25:55 - also exist for utilization behavior for instance utilization behavior
  • fast_forward00:25:58 - patients they are not really able to inhibit their
  • fast_forward00:26:01 - motor programs for grasping objects and uh just
  • fast_forward00:26:04 - reacting in a very uh quick way to what they say so this uh this tells a story
  • fast_forward00:26:09 - on on how uh this more um abstract thinking part is tied to the sensory motor
  • fast_forward00:26:16 - programs okay but now this these this this in how do you what's What's the role
  • fast_forward00:26:21 - of the internal simulation in that case?
  • fast_forward00:26:23 - Because on the one hand, you're saying, well, we run these internal simulations.
  • fast_forward00:26:27 - Apparently, they would run in parallel.
  • fast_forward00:26:29 - But on the other hand, if you look at the deficits that you mentioned in these
  • fast_forward00:26:33 - patients, it's not that they are executing a whole sequence of actions in parallel, right?
  • fast_forward00:26:41 - In utilization behavior, you would grasp a single object and the whole organism
  • fast_forward00:26:45 - would be focused on that.
  • fast_forward00:26:46 - So therefore, is this kind of evidence coming more from the clinic,
  • fast_forward00:26:53 - from clinical studies, really supporting this hypothesis of internal assimilation
  • fast_forward00:26:57 - and the role of action in the structure of cognition?
  • fast_forward00:26:59 - Well, for the first evidence I mentioned of the patient who was unable to inhibit
  • fast_forward00:27:05 - the imagined action, that's quite clear, because then what you think is what you do.
  • fast_forward00:27:11 - So if you imagine, let's say, pointing to one direction, then you point in that
  • fast_forward00:27:16 - direction, you cannot really stop from doing that.
  • fast_forward00:27:18 - So in this framework that I'm supporting, the idea is that you run this imaginary
  • fast_forward00:27:23 - simulation of pointing.
  • fast_forward00:27:25 - This is typically done by also inhibiting the overt motor execution.
  • fast_forward00:27:30 - But if you have a deficit, you cannot inhibit that output.
  • fast_forward00:27:33 - It rapidly turns out into an overt action.
  • fast_forward00:27:36 - In the utilization behavior, the link is tiny in a sense.
  • fast_forward00:27:41 - The idea is that one idea in the literature of affordances, also the canonical
  • fast_forward00:27:48 - neurons literature, is that when you look at an object,
  • fast_forward00:27:52 - then you automatically, let's say, pre-potenciate some motor programs that are
  • fast_forward00:27:58 - good for interacting with that object.
  • fast_forward00:28:00 - Jet okay but typically this does not result into an overt execution so in a
  • fast_forward00:28:07 - sense this is just a pre potentiation which can be also interpreted in terms
  • fast_forward00:28:12 - of simulating grasping this cap in front of me,
  • fast_forward00:28:16 - So in that case, I'm not reasoning about simulating the action.
  • fast_forward00:28:18 - This is just a more automatic process of mentally rehearsing of the good action that they can do.
  • fast_forward00:28:25 - Again, if you don't have this, let's say, this mechanism working well,
  • fast_forward00:28:31 - you cannot really inhibit also the overt execution.
  • fast_forward00:28:35 - I would say that while in my first example of the imaginary situation,
  • fast_forward00:28:39 - the link is clearer, in this second example, it's less clear.
  • fast_forward00:28:42 - But to me, it's another supporting example that your internal mental process
  • fast_forward00:28:47 - is always oriented towards anticipating possibilities for action.
  • fast_forward00:28:51 - In some cases, this is just a simple automatic process that tells you what is possible to do.
  • fast_forward00:28:56 - In other cases, this is more intentional imagery.
  • fast_forward00:29:01 - So you really run long-term simulation imaging sequences of actions.
  • fast_forward00:29:06 - But I see a continuity in this.
  • fast_forward00:29:09 - So the good thing about this framework is that you do not have to postulate
  • fast_forward00:29:13 - any different set of cognitive representation for different tasks,
  • fast_forward00:29:17 - any module for thinking that is completely segregated from the action control and specification,
  • fast_forward00:29:23 - but you have a continual reuse of the same abilities in more complex ways.
  • fast_forward00:29:28 - Okay, but then would you see this as being, let's say, layered in some way,
  • fast_forward00:29:33 - that you have, let's say, sensory motor capabilities of, let's say,
  • fast_forward00:29:37 - varying levels of complexity with some sort of discrete steps between them,
  • fast_forward00:29:41 - or is it really a continuum?
  • fast_forward00:29:44 - Well, that's a hard question. one important
  • fast_forward00:29:49 - thing in this framework that probably answers partially to
  • fast_forward00:29:51 - your question is that we have
  • fast_forward00:29:54 - to probably focus we have to talk about
  • fast_forward00:29:57 - actions as the unity and also we
  • fast_forward00:30:00 - know that action can be specified at different levels of detail
  • fast_forward00:30:03 - so there are actions that are specified
  • fast_forward00:30:06 - at the level of movement of a single finger whereas there
  • fast_forward00:30:09 - are other actions in which the action but also
  • fast_forward00:30:12 - the goal of the action is specified at a more abstract
  • fast_forward00:30:15 - level that is grasping this uh this object
  • fast_forward00:30:18 - this cup whereas there is still another level of
  • fast_forward00:30:21 - action and intention associated which is uh let's say drinking from this cup
  • fast_forward00:30:27 - so the actions uh all these all these uh levels are are in a sense they they
  • fast_forward00:30:35 - are they support one another in a sense so of Of course,
  • fast_forward00:30:39 - the more abstract actions have to be finally specified in more fine-grained
  • fast_forward00:30:46 - terms of the movements of the fingers.
  • fast_forward00:30:48 - But this is a structure, a cognitive structure that has different layers probably.
  • fast_forward00:30:53 - Or we don't know exactly, but at least you have a big structure in which you
  • fast_forward00:30:59 - can really specify actions at different levels of complexity,
  • fast_forward00:31:04 - a different level of abstraction with more complex intentions.
  • fast_forward00:31:06 - And the arguments that we do is not that we always use the lowest level,
  • fast_forward00:31:12 - the lowest possible levels.
  • fast_forward00:31:13 - At some point, you can really think and do some imagery by using actions and
  • fast_forward00:31:19 - their associated intentions at intermediate or even at abstract levels.
  • fast_forward00:31:24 - So that's probably answers. Right. So now one way to test some of these ideas,
  • fast_forward00:31:29 - you were performing and presenting experiments on joint action.
  • fast_forward00:31:33 - Yes. So how does joint action now help you to understand this model of cognition and action in the end?
  • fast_forward00:31:43 - Well, the reason why I went into joint action in my presentation is that, as I told before,
  • fast_forward00:31:51 - I would like to come out with a convincing story or more complex cognitive abilities
  • fast_forward00:31:55 - developed based on simpler cognitive abilities.
  • fast_forward00:31:59 - So here the steps that I presented are, okay, I start doing my action,
  • fast_forward00:32:04 - but then I am in a social domain, and so maybe I need to do some action together with you.
  • fast_forward00:32:12 - So in this case, I can reuse a lot of what I know about my action system to
  • fast_forward00:32:17 - anticipate you and to get into your intentions.
  • fast_forward00:32:21 - So to get rid of your motor system or your motor action of your intentions.
  • fast_forward00:32:26 - In this case, this is just the beginning of a story in which I reuse my sensory
  • fast_forward00:32:32 - motor knowledge or my action execution and prediction abilities to get into
  • fast_forward00:32:38 - more and more complex situations.
  • fast_forward00:32:40 - In this case, it is a social situation, but it could be even a non-social situation.
  • fast_forward00:32:45 - Still another step in this passage is that, okay, now I'm able to use my predictive
  • fast_forward00:32:52 - abilities to predict you.
  • fast_forward00:32:54 - I use my body and my action system as a model to understand your body and your action system.
  • fast_forward00:32:59 - And then I maybe use these abilities to plan long-term joint action.
  • fast_forward00:33:05 - To do that then i have planned some more coordinating actions
  • fast_forward00:33:08 - so i have to plan how to really achieve them in practice i
  • fast_forward00:33:11 - have maybe to to invent some way to better
  • fast_forward00:33:14 - coordinate with you and maybe at some
  • fast_forward00:33:17 - point you invent more and more complex things you extend the
  • fast_forward00:33:20 - boundaries of your control to the control of my body or
  • fast_forward00:33:23 - to the control of our body or our combined actions
  • fast_forward00:33:27 - and maybe also to the control of your internal
  • fast_forward00:33:30 - states so your beliefs your intentions so
  • fast_forward00:33:33 - i do some action to change your intention so not only
  • fast_forward00:33:36 - i control my body i now control me and you and i
  • fast_forward00:33:38 - control you and i know that uh even after
  • fast_forward00:33:41 - this interaction you will have some new beliefs i extend my control abilities
  • fast_forward00:33:46 - control capabilities from myself to you so that that's uh this the very beginning
  • fast_forward00:33:52 - of a story in which we begin to to go into more and more complex cognitive abilities
  • fast_forward00:33:58 - is starting from simpler ones. Right.
  • fast_forward00:34:02 - So for the joint action experiment, you were looking at hand movements essentially. Right.
  • fast_forward00:34:09 - You're tracking hand mostly yes and then you
  • fast_forward00:34:12 - also build a model of that yes okay so so
  • fast_forward00:34:16 - which aspect now of the joint action control of
  • fast_forward00:34:20 - hands do you capture in this in this model well if we think the model specifically
  • fast_forward00:34:28 - then uh well in a sense the key idea of the model is that it's not completely is not new.
  • fast_forward00:34:37 - There are many similar models, but the idea in a sense is that I really use
  • fast_forward00:34:41 - my own action abilities as a model to understand your action right now.
  • fast_forward00:34:47 - So the model can predict for instance the hand trajectory, but at the same time
  • fast_forward00:34:54 - it can also give some hint into your intentions and goals.
  • fast_forward00:34:59 - Because I know what are the goals that are linked to my model actions.
  • fast_forward00:35:03 - How does the model know that? Well, let's imagine I see you moving your arm towards a cup, okay?
  • fast_forward00:35:11 - So now the hypothesis is that I run an internal simulation of the possible actions
  • fast_forward00:35:15 - that I could be that are quite compatible with the actions that I'm seeing.
  • fast_forward00:35:20 - I start predicting better and better your trajectory, and then let's see that
  • fast_forward00:35:24 - this model fits quite well the data, so your hand trajectory.
  • fast_forward00:35:28 - Because I know what is the goal
  • fast_forward00:35:29 - of my hand action, I also can have some hypothesis of what is the goal.
  • fast_forward00:35:34 - So because I know that my trajectory eventually reaches the cap and then grasps
  • fast_forward00:35:38 - it, then I can also infer or at least hypothesize that you will do the same.
  • fast_forward00:35:43 - So I also know what is your goal.
  • fast_forward00:35:45 - That's the key idea. But then the intentional labeling of the sequence is then
  • fast_forward00:35:53 - with reference to your own action or that is in reference in some way to observing the other?
  • fast_forward00:36:00 - So the idea of these kind of models is that you use what you know about the
  • fast_forward00:36:06 - link between the trajectory of your arms and your goal. You know them because you execute them.
  • fast_forward00:36:11 - You use exactly the same link to get some understanding of what the other is doing.
  • fast_forward00:36:17 - So that's the idea. So you have first some sort of self-monitoring system that
  • fast_forward00:36:21 - develops a self-model that you then generalize to the other.
  • fast_forward00:36:24 - This would be the step. Exactly. So I will not necessarily call it a self-monitoring
  • fast_forward00:36:29 - in that you simply need a forward model and an inverse model.
  • fast_forward00:36:33 - So you need some control capabilities for yourself.
  • fast_forward00:36:37 - And then you transfer, you use this exactly as a model to understand the other.
  • fast_forward00:36:42 - So that's more or less the key idea. And this idea is pursued by many people
  • fast_forward00:36:46 - actually also. Right. Yeah. Okay.
  • fast_forward00:36:51 - So what's, do you feel confident then that this original idea you had about
  • fast_forward00:36:55 - this more, let's say, embodied problem solving, action as grounding, cognition,
  • fast_forward00:36:59 - what tells you in this joint action task where you now are generalizing this
  • fast_forward00:37:04 - model to social interaction, that this is working?
  • fast_forward00:37:07 - Because in some sense, I could also argue, well, it works because you have been
  • fast_forward00:37:10 - imposing, you have really made the task pretty abstract, right?
  • fast_forward00:37:13 - Because I'm just observing these hands moving in one plane. plane,
  • fast_forward00:37:18 - I don't have to do anything about, let's say, invariant recognition of posture, of limbs and so on.
  • fast_forward00:37:24 - So to really now generalize this to, let's say, a realistic scenario where you
  • fast_forward00:37:30 - have to observe another human moving about in space, is that just a small matter
  • fast_forward00:37:35 - of programming or are there some fundamental steps still missing?
  • fast_forward00:37:40 - Well, to answer this question, probably we have to go into a bigger picture.
  • fast_forward00:37:44 - The bigger picture to me is that now, although I have emphasized these motor
  • fast_forward00:37:50 - predictions, that's not the whole story.
  • fast_forward00:37:53 - In a sense, my opinion is that the brain is a smart guy.
  • fast_forward00:37:56 - The brain always uses all the knowledge that it has, at least in principle, to do the job.
  • fast_forward00:38:03 - The point here is that in more realistic joint action or action observation
  • fast_forward00:38:08 - scenarios, there is a lot of information
  • fast_forward00:38:10 - available. There is perceptual information, which is available.
  • fast_forward00:38:15 - There is some context information, such as, for instance, I see what are the
  • fast_forward00:38:18 - objects within your pre-personal space. So that's my prior.
  • fast_forward00:38:23 - I can have a lot of prior information on you. For instance, I know that you like very much the beer.
  • fast_forward00:38:31 - So if there are in front of you, I see some beer and I see some Coke,
  • fast_forward00:38:35 - I would maybe anticipate that you will grasp the beer.
  • fast_forward00:38:39 - And then I also use this motor simulation that I told, because it is also useful.
  • fast_forward00:38:46 - So I'm not assuming that the brain only uses this motor simulation part.
  • fast_forward00:38:50 - It uses the motor simulation to support this action understanding,
  • fast_forward00:38:55 - because in this context, it is very salient and very useful.
  • fast_forward00:39:00 - Why is it very useful? Because, well, I have a very good model of my body.
  • fast_forward00:39:05 - So it's a very good model, very good predictive model for your movements.
  • fast_forward00:39:08 - Of course, I have also some purely perceptual predictive mechanisms that can help me predicting you.
  • fast_forward00:39:15 - But I would say that in the case of human understanding, the model of myself
  • fast_forward00:39:21 - is a very, very good model of you in most cases.
  • fast_forward00:39:24 - So this is what the brain will use the most.
  • fast_forward00:39:28 - But if you go into the biggest picture, you can think of it as a big Bayesian
  • fast_forward00:39:32 - process happening. So in a Bayesian process, you basically use all the information
  • fast_forward00:39:36 - you have, information which comes first, then can be also used as a prior to
  • fast_forward00:39:41 - run the rest of the process.
  • fast_forward00:39:42 - And then all the new information that is collected is fused in an intelligent way.
  • fast_forward00:39:48 - Intelligent here means that it is weighted depending on its uncertainty.
  • fast_forward00:39:52 - So if the motor system is a very good model, the information it provides will be used very much.
  • fast_forward00:39:59 - Much if this information is not good or it is failing then it will not be used
  • fast_forward00:40:04 - very much yeah but you seem to now to drift away a little bit from your original.
  • fast_forward00:40:10 - Proposal because i thought you started out by saying look cognition is really
  • fast_forward00:40:15 - predicated on action yeah right but now you seem to say something like well
  • fast_forward00:40:19 - motor the motor system is one of many sources of information you could consider
  • fast_forward00:40:24 - because the bayesian system basically use all the knowledge it has,
  • fast_forward00:40:27 - and it has knowledge about the motor system, but also about,
  • fast_forward00:40:30 - let's say, perceptual systems, or from memory.
  • fast_forward00:40:35 - So aren't you a bit drifting now in that argument?
  • fast_forward00:40:38 - Well, I don't think so, because as I told before, the way you collect this information
  • fast_forward00:40:44 - in the first place is through learning.
  • fast_forward00:40:46 - And through learning, the motor system has really influenced you a lot.
  • fast_forward00:40:51 - So also through learning, this perceptual information that I have,
  • fast_forward00:40:55 - I have acquired it by also interacting with the world.
  • fast_forward00:40:58 - So the motor system really is used very much not only online,
  • fast_forward00:41:02 - as I told now, but also during learning.
  • fast_forward00:41:05 - Of course, we don't have to be radical on that. So we don't think that only
  • fast_forward00:41:10 - the motor system does something in the brain.
  • fast_forward00:41:13 - There are many parts of the brain that are used.
  • fast_forward00:41:16 - And probably the point is slightly different.
  • fast_forward00:41:20 - The point is not all about the motor system. So the point is that the ability
  • fast_forward00:41:24 - that you have to interact with the external world,
  • fast_forward00:41:27 - which can be supported by many systems in the brain,
  • fast_forward00:41:30 - prominently by the motor system, but well many system
  • fast_forward00:41:33 - you reuse them consistently to achieve new
  • fast_forward00:41:37 - cognitive abilities so to develop and and
  • fast_forward00:41:40 - then to achieve new new goals specified at different levels okay but um so then
  • fast_forward00:41:45 - you're saying well action let's say impregnates all other aspects yes of of
  • fast_forward00:41:50 - cognition and perception through the cause of learning okay and this so it has
  • fast_forward00:41:54 - a direct and otherwise an indirect impact yeah yeah Yeah, there are both.
  • fast_forward00:41:59 - I would say that the first one is through learning. The second one is during
  • fast_forward00:42:03 - overt interaction with the external world.
  • fast_forward00:42:05 - And then through the internalization of this action and prediction process,
  • fast_forward00:42:10 - you can imagine new scenarios.
  • fast_forward00:42:12 - But this doesn't mean that when you can have some motor simulation,
  • fast_forward00:42:16 - you close your eyes, you don't use percentile information.
  • fast_forward00:42:19 - That would be foolish from the point of view of the brain. and and
  • fast_forward00:42:22 - there is an important uh point of this view
  • fast_forward00:42:25 - is that uh as you know in action understanding
  • fast_forward00:42:28 - there are many views one view is that you
  • fast_forward00:42:31 - i call this the mirrorless view is that
  • fast_forward00:42:34 - the first thing that is active is the recognition of the goal then the recognition
  • fast_forward00:42:38 - of the goal in the mirror system of course it eventually helps also predicting
  • fast_forward00:42:43 - the action there is a second view which is related but in some sense also very
  • fast_forward00:42:49 - different because this view is that what you do first is prediction.
  • fast_forward00:42:53 - So it is prediction which comes first and prediction helps then recognizing the goal.
  • fast_forward00:42:59 - But still another view is that all these motor simulation things,
  • fast_forward00:43:04 - they are not so important because we really use much more abstract knowledge
  • fast_forward00:43:09 - of what the other person should do, given the context.
  • fast_forward00:43:12 - So that's more theological knowledge, if you want.
  • fast_forward00:43:15 - Well, in the view I'm supporting, it is not so useful to say what is more used and what is less used.
  • fast_forward00:43:24 - So what comes first can be used as a prior for the rest of the things.
  • fast_forward00:43:28 - And what is more reliable is used more than the other things.
  • fast_forward00:43:33 - It happens, at least to me, that
  • fast_forward00:43:35 - this motor simulation in that specific context, they are very reliable.
  • fast_forward00:43:39 - This is why I think they play a prominent role, not because of some categorical distinctions.
  • fast_forward00:43:45 - And also relating to this issue of what comes first, goal recognition or prediction,
  • fast_forward00:43:50 - that's really depending on the task.
  • fast_forward00:43:52 - So if I have a big prior knowledge of what the most likely goals will be,
  • fast_forward00:43:56 - then it is probable that I will force some hypothesis or some probability distribution
  • fast_forward00:44:01 - over those goals much before starting predicting you.
  • fast_forward00:44:06 - But if I cannot see what are the goals, let's say the objects you can grasp,
  • fast_forward00:44:11 - I will probably start by predicting and by prediction I will get into some hypothesis of what is your goal.
  • fast_forward00:44:18 - So everything can be very flexible.
  • fast_forward00:44:21 - Okay, but you are saying ideally you take a model-based approach towards yes
  • fast_forward00:44:26 - in this case also social perception you want to say yes you want to get to the
  • fast_forward00:44:31 - goal of the intentional state of the other agent as quickly as possible and
  • fast_forward00:44:36 - from there you just reconstruct the rest of the action,
  • fast_forward00:44:41 - yeah yes in some sense so yeah.
  • fast_forward00:44:45 - Typically, I want to go into the goal when this is useful for my task.
  • fast_forward00:44:49 - So if my task is understanding which cup will you grasp, that's the goal information that I need.
  • fast_forward00:44:57 - Whereas if I don't want to hurt you, it's more the trajectory that I want to know.
  • fast_forward00:45:01 - So for example, when we are driving, I don't really need to know where are you going.
  • fast_forward00:45:09 - I don't care. I don't want to cross you or to hurt you. So it's not so much
  • fast_forward00:45:15 - the goal of what you're doing that is important, but it is the trajectory that you are doing right now.
  • fast_forward00:45:22 - Of course, also the goal is interesting because the goal tells me a lot about your trajectory.
  • fast_forward00:45:27 - So I would say that what you ask, what you want to infer is very much task dependent.
  • fast_forward00:45:36 - And as a side effect, you will also infer many other things that give you useful
  • fast_forward00:45:42 - priors or useful hints. experiments. So that's my view.
  • fast_forward00:45:46 - You don't only run experiments, right? To test these ideas, you also use robots.
  • fast_forward00:45:52 - Yes. So how have robots helped you to make progress on these issues?
  • fast_forward00:45:58 - Well, yes, we are running many computational and also robotic experiments.
  • fast_forward00:46:05 - So So one thing that we have recently seen is that probably this is very interesting
  • fast_forward00:46:14 - again for the argument that more and more complex cognitive abilities can be
  • fast_forward00:46:18 - developed on top of this sensory-motor interaction.
  • fast_forward00:46:20 - One thing that we are investigating with computational models and robots is
  • fast_forward00:46:24 - the ability to signal in social context.
  • fast_forward00:46:27 - So just to explain this very quickly, the point is that if we are supposed to
  • fast_forward00:46:32 - do one joint action okay let's say building some tower together but of red and
  • fast_forward00:46:39 - blue blocks okay one red one blue one red one blue okay now we are interacting
  • fast_forward00:46:44 - we are supposed to do that,
  • fast_forward00:46:46 - and we are able to coordinate to do that but now let's imagine only i know.
  • fast_forward00:46:53 - What is the goal? What is the tower to be built? You don't know.
  • fast_forward00:46:56 - So one thing that we are investigating with robots is how we come out with some
  • fast_forward00:47:00 - good coordination with only one person knowing the task.
  • fast_forward00:47:05 - In that case, without using overt or linguistic communication.
  • fast_forward00:47:10 - So what we have seen also in robot, that's a very hard problem.
  • fast_forward00:47:14 - Because when you run computational robotic models, then you put your ideas into practice practice,
  • fast_forward00:47:19 - and you see that by simply predicting the action of the others doesn't work
  • fast_forward00:47:23 - too much because then I start predicting you,
  • fast_forward00:47:27 - but so the guy who doesn't know the job has to predict the actions of the other,
  • fast_forward00:47:32 - but it is always one step later, it's too late.
  • fast_forward00:47:36 - So it doesn't work very well. So we come out with the idea that the guy who
  • fast_forward00:47:41 - knows the job can also in some sense support the predictive processes of the
  • fast_forward00:47:46 - other or help the other in some sense.
  • fast_forward00:47:49 - So how would I help my other robot? Well, one simple example is that I can make
  • fast_forward00:47:56 - my behavior more predictable, or I can make my behavior more informative for you.
  • fast_forward00:48:03 - Such as I can use my choice of action to letting you understand very well what are my intentions.
  • fast_forward00:48:11 - So, you typically try to understand my intentions, but if the uncertainty is
  • fast_forward00:48:15 - too high, I can in some sense help you with the wise choice of actions.
  • fast_forward00:48:21 - That can be very intuitive.
  • fast_forward00:48:23 - But it could also be a way to test hypotheses about the other.
  • fast_forward00:48:26 - It's not just making your behavior more predictable. It must be a way to test
  • fast_forward00:48:30 - whether the other has the right model of you.
  • fast_forward00:48:32 - Absolutely. Yes, that's the idea. So, we design… But it's not the same thing, right? No, no, no.
  • fast_forward00:48:37 - Well, they are related. So, in one sense,
  • fast_forward00:48:42 - Also, in my example, let's call the guy who has the knowledge the leader and the other the follower.
  • fast_forward00:48:48 - So in a sense, the leader can have some uncertainty on what are the models of the follower.
  • fast_forward00:48:55 - So, of course, he can use some actions and monitor the reactions of the other
  • fast_forward00:49:02 - person just to know what are the good models, what are the models of the other
  • fast_forward00:49:08 - agents, what is he doing.
  • fast_forward00:49:10 - Right. And when the leader infers that the follower does not have good models, then he will fail.
  • fast_forward00:49:17 - Then he can, in a sense, try to help the follower.
  • fast_forward00:49:22 - So the two processes are interconnected.
  • fast_forward00:49:25 - So by monitoring the process and by always trying to keep track of the uncertainty
  • fast_forward00:49:32 - of the other person, you really can plan helping actions. Right.
  • fast_forward00:49:37 - But now the thing was interesting that, so we go from action now to interaction.
  • fast_forward00:49:41 - Yes. And in your research plan, that's really a very crucial step, right?
  • fast_forward00:49:48 - Because you really seem to think that or propose that it's by getting a handle
  • fast_forward00:49:53 - on interaction that you really can get to, let's say, a more generalized understanding
  • fast_forward00:49:56 - of social behavior and also cultures.
  • fast_forward00:50:00 - Yeah. So how should I see that generalization exactly?
  • fast_forward00:50:05 - The point is that I think that for many disciplines, such as the study of language
  • fast_forward00:50:11 - and communication, the natural starting point would be naturalistic joint action.
  • fast_forward00:50:16 - That will be a nice starting point because now the hypothesis is that the mechanism
  • fast_forward00:50:21 - that we use for interacting with the others in sensory motor domains,
  • fast_forward00:50:26 - then they provide a scaffold.
  • fast_forward00:50:28 - So they provide some help for developing, let's say, for instance,
  • fast_forward00:50:34 - linguistic communication.
  • fast_forward00:50:35 - The hypothesis, which has been called by Levinson this more interaction engine hypothesis,
  • fast_forward00:50:41 - is that by simply interacting in the external world, we really have some strong
  • fast_forward00:50:47 - pragmatic abilities to infer the action of the other, to do some truth-taking,
  • fast_forward00:50:52 - to do some joint attention,
  • fast_forward00:50:53 - to anticipate the action and the intentions of the others.
  • fast_forward00:50:56 - And this is a strong universal basis also for language communication.
  • fast_forward00:51:01 - So, he hypothesizes, for instance, the children use this interaction engine
  • fast_forward00:51:05 - to develop, so to understand and to learn their natural language.
  • fast_forward00:51:12 - And that's exactly a point that we want to make.
  • fast_forward00:51:16 - So we started studying this kind of joint actions. Now we come out with a formulation
  • fast_forward00:51:21 - of the problem in which the two persons happen to solve a joint action optimization process.
  • fast_forward00:51:27 - This joint action optimization process means that it is not only so I don't
  • fast_forward00:51:33 - have to care only about my action, but also the joint outcome of the action.
  • fast_forward00:51:38 - Because I have to care about the joint action and the joint outcome of our action,
  • fast_forward00:51:43 - which is a joint goal, then I also have to care about you.
  • fast_forward00:51:46 - This is why in some cases, as I told before, I help you, not because I'm altruistic.
  • fast_forward00:51:52 - I'm helping you because by helping you, I help the joint goal.
  • fast_forward00:51:54 - All. So I'm sorry but I'm not so much altruistic, that's all individualistic
  • fast_forward00:51:59 - but at the same time I think that this helping action,
  • fast_forward00:52:06 - is is really the first primitive form of a
  • fast_forward00:52:09 - communicative action because as i told before i
  • fast_forward00:52:12 - can make my action more observable by
  • fast_forward00:52:15 - you more predictable by you more understandable or more
  • fast_forward00:52:18 - diagnostic for you diagnostic here simple
  • fast_forward00:52:21 - means simply means that if you have two hypotheses on what i'm doing i make
  • fast_forward00:52:25 - my action uh let's say for instance i exaggerate the movement such as to make
  • fast_forward00:52:29 - you understand very quickly what i what is my right intentions that's diagnostic
  • fast_forward00:52:34 - but that That exaggeration that I do, it's really a form of communication.
  • fast_forward00:52:40 - That's really the bootstrap also of all other forms of communication,
  • fast_forward00:52:44 - including linguistic communication.
  • fast_forward00:52:46 - So the general hypothesis here, this is why I'm interested in this passage from
  • fast_forward00:52:51 - joint action to more complex form of cognition, is that by interacting,
  • fast_forward00:52:56 - because there are some constraints,
  • fast_forward00:52:58 - some joint constraints that we have to fulfill, feel
  • fast_forward00:53:01 - then in this joint optimization
  • fast_forward00:53:04 - framework it's quite obvious that i have
  • fast_forward00:53:07 - to do something also for you for your sake right and
  • fast_forward00:53:10 - in our formulation that that that means that in addition to having some motor
  • fast_forward00:53:15 - intention we also have some communicative intention the communicative intention
  • fast_forward00:53:20 - is has a cost because for instance for exaggerating the movement i pay a cost
  • fast_forward00:53:26 - in terms of biomechanical a cost of something.
  • fast_forward00:53:28 - So such as, for instance, when I am in a noisy pub, I have to over-articulate.
  • fast_forward00:53:34 - Or if you think the mother is, for example, when the child directed speech,
  • fast_forward00:53:39 - so you exaggerate, that's done for the sake of the other person.
  • fast_forward00:53:43 - Ultimately, we say that's done for the sake of achieving a good joint goal,
  • fast_forward00:53:47 - which is understanding one another.
  • fast_forward00:53:49 - But that's really the first form of motor of communicative intention that I have.
  • fast_forward00:53:55 - So it's for you, it's really the signaling mechanisms that would allow the bootstrapping
  • fast_forward00:53:59 - of communication and language, right?
  • fast_forward00:54:02 - But that's still a project that will be realized in the future.
  • fast_forward00:54:06 - Yes, but I would say that the important component is always the missing one
  • fast_forward00:54:11 - from the current studies, which is the more pragmatic component of language.
  • fast_forward00:54:15 - So the pragmatics of language as the early pragmatists, which study language know very, very well.
  • fast_forward00:54:21 - So the pragmatic component is prominent because then the ground ego symbols,
  • fast_forward00:54:25 - the grammar, of course, they are hard problems, but the pragmatics,
  • fast_forward00:54:28 - they are the most important part, such as for instance, knowing Knowing what requesting means,
  • fast_forward00:54:35 - knowing what telling means or promising means or all these joint action and joint attention,
  • fast_forward00:54:44 - tour-taking mechanism, all of them provide a big scaffolding for the emergence
  • fast_forward00:54:49 - of linguistic communication. Right.
  • fast_forward00:54:51 - So I got that. But now, so to finish up, two questions.
  • fast_forward00:54:57 - So you have this really big ambitious program, and in the end,
  • fast_forward00:55:01 - you'll generate a whole of culture from action, which is going to be still a
  • fast_forward00:55:07 - long trajectory, but you're well on your way.
  • fast_forward00:55:10 - But in doing this, what would be Giovanni's Law?
  • fast_forward00:55:14 - Giovanni's law? Okay. Well, my law would be maybe starting understanding cognitions
  • fast_forward00:55:22 - from the primitive equipment of our ancestors.
  • fast_forward00:55:27 - So that would be more an evolutionary route towards the more complex cognitive skills.
  • fast_forward00:55:33 - Well, I can probably give three
  • fast_forward00:55:35 - examples that are very simple in the three domains that we target now.
  • fast_forward00:55:39 - One is the social domain that we have discussed.
  • fast_forward00:55:42 - So you start from controlling your body, your actions, to controlling our actions,
  • fast_forward00:55:47 - to controlling then your mental states.
  • fast_forward00:55:49 - That's communication in a sense. Communication is really putting something into your brain.
  • fast_forward00:55:53 - And then you also develop culture and pedagogy, for instance,
  • fast_forward00:55:57 - for controlling that at a very, very longer timescale.
  • fast_forward00:56:00 - So by developing some culture, the human beings can really control the behavior
  • fast_forward00:56:05 - of many people over millennia probably, and also by pedagogy.
  • fast_forward00:56:09 - So that's in the social domain.
  • fast_forward00:56:11 - You can see this also in the control of the external world, on this environmental
  • fast_forward00:56:16 - scaffolding or the environmental shaping.
  • fast_forward00:56:19 - So we start from the control of our body to the control maybe of the reality.
  • fast_forward00:56:25 - Our peripersonal space with some objects and then
  • fast_forward00:56:28 - you start controlling tools for instance
  • fast_forward00:56:31 - for doing more complex actions and but then
  • fast_forward00:56:34 - tools you have also to build tools maybe at some
  • fast_forward00:56:37 - point you build the city of barcelona so then in a sense
  • fast_forward00:56:40 - you extend the boundaries of your control from the control your body to the
  • fast_forward00:56:44 - control of the whole environment we humans really modify our environment very
  • fast_forward00:56:48 - much such that then they support much better our cognition so that's another
  • fast_forward00:56:53 - story not only in the social domain also in the environmental domain the third
  • fast_forward00:56:57 - the third and last one is in the more,
  • fast_forward00:57:00 - thought domain or the cognitive control domain so you just start from controlling
  • fast_forward00:57:04 - your action and then you end up in some way by controlling your thought processes
  • fast_forward00:57:10 - and by controlling your cognition over long long long time scales for instance
  • fast_forward00:57:15 - one example that i do well,
  • fast_forward00:57:17 - let's imagine i want to become a prominent scientist
  • fast_forward00:57:21 - which apparently doesn't doesn't happen right
  • fast_forward00:57:24 - now so i have to control my behavior for 20 years
  • fast_forward00:57:27 - maybe so the way i do that is by well by cognitive control in a sense by setting
  • fast_forward00:57:33 - some goal states into my mind then then control my behavior on longer longer
  • fast_forward00:57:37 - longer time scales then again you have this control of the body the or the control
  • fast_forward00:57:42 - of my short-term actions that become the control of my entire life.
  • fast_forward00:57:48 - I have given three examples of how this research program, I would say,
  • fast_forward00:57:53 - would start from very simple sensory motor abilities to very,
  • fast_forward00:57:57 - very complex abilities that cross the boundaries of many disciplines.
  • fast_forward00:58:02 - Sociology on the one hand, and let's say architecture on the other hand,
  • fast_forward00:58:05 - or maybe the study of consciousness on the other hand.
  • fast_forward00:58:08 - And you have even integrated it in your own career planning.
  • fast_forward00:58:12 - Ah, of course. so the last thing the very last question then is prediction so
  • fast_forward00:58:16 - if we come back then five years from now as a famous scientist to the summer school again.
  • fast_forward00:58:23 - So what's the one prediction we can test on you by then?
  • fast_forward00:58:27 - So what's the one prediction you would like to make today that you feel very strongly about?
  • fast_forward00:58:31 - And we can ask you about five years from now.
  • fast_forward00:58:34 - Okay, so about the future of this field or about the… Your research program. Okay.
  • fast_forward00:58:41 - So about my research program. Well, I think I elucidated the principle,
  • fast_forward00:58:46 - my key goal, which is understanding higher cognition, how our cognition is based
  • fast_forward00:58:52 - on sensory motor and predictive abilities.
  • fast_forward00:58:55 - So the prediction is that at least we will have some nice demonstration in these
  • fast_forward00:58:59 - three fields that I've mentioned before, much stronger demonstration than the ones that we have now.
  • fast_forward00:59:04 - And from that, we can really try to build an understanding of how the brain
  • fast_forward00:59:09 - implements higher cognitive skills.
  • fast_forward00:59:11 - So at least one demonstration for each of the three fields. That's very good.
  • fast_forward00:59:16 - So Giovanna Pizzuto, thank you very much for this conversation. Thank you, Paul.
  • fast_forward00:59:22 - The csn podcast was produced by the convergent science network of biometrics
  • fast_forward00:59:28 - and biohybrid systems a project funded by the european sevens research framework program.
  • fast_forward00:59:36 - Music.

Be the first to leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

Convergent PozitivLinia Logo

Exploring the convergence of neuroscience, robotics, and AI through conversations with leading researchers since 2010.

A project of the Convergent Science Network Foundation.

© CSN Podcasts. Developed by IMCreativeWEBC

0%

Login to enjoy full advantages

Please login or subscribe to continue.

Go Premium!

Enjoy the full advantage of the premium access.

Stop following

Unfollow Cancel

Cancel subscription

Are you sure you want to cancel your subscription? You will lose your Premium access and stored playlists.

Go back Confirm cancellation