Listeners:

Top listeners:

00:00 00:00

Go to album

Paul Verschure on consciousness and distributed adaptive control CSN Podcasts
Edvard Moser on grid cells and entorhinal cortex CSN Podcasts
Giacomo Rizzolatti on mirror neurons and action understanding CSN Podcasts
Robert Axelrod on game theory and prisoner's dilemma CSN Podcasts
Adrian Owen on disorders of consciousness and vegetative state CSN Podcasts
Jonathan Whitlock on markerless motion capture and posterior parietal cortex CSN Podcasts
Luis Puelles on neuroanatomy and prosomeric model CSN Podcasts
Zoltan Molnar on subplate neurons and cortical development CSN Podcasts

Lars Muckli on predictive processing and visual cortex

Episode 7 15.03.2019

PLAY EPISODE

Season 2019

0 Followers 13 Episodes

Description

Does the brain see the world or predict it? Visual neuroscientist Lars Muckli presents evidence that early visual cortex receives top-down predictive signals from higher areas, challenging the textbook view of vision as a purely bottom-up feature extraction process and raising hard questions about where prediction ends and perception begins. Subscribe for more from the Convergent Science Network podcast series. Lars Muckli joins Paul Verschure and Tony Prescott to explain how apparent motion, one of the simplest visual illusions, became a window into the predictive architecture of the visual brain. Using fMRI with retinotopic mapping, Muckli’s lab discovered that the space between two alternating dots is filled with neural activity that cannot be explained by local V1 processing alone. EEG experiments revealed that motion-sensitive area V5 responds approximately 40 milliseconds before retinotopic V1 regions, and TMS applied to V5 before stimulus onset eliminates the predictability effect on the apparent motion trace , both pointing to a feedback signal carrying predictive information. The conversation becomes a rigorous methodological interrogation. Verschure challenges whether the data truly require a hierarchical predictive model or could be explained by lateral interactions within V1, where 97 percent of synapses originate locally. Muckli acknowledges that lateral and top-down contributions likely combine, proposing a model where higher areas provide a coarse motion envelope while local V1 circuitry adds spatial precision. Layer-specific fMRI analysis of occluded scene regions shows predictive content distributed across cortical layers rather than confined to specific laminae, suggesting the implementation of prediction in cortical circuits may be more distributed than canonical models assume. Key topics include why the predictive processing framework offers a more parsimonious account of visual processing than feedforward hierarchies, the methodological challenges of distinguishing prediction from postdiction, what layer-specific fMRI reveals about cortical feedback, and whether the predictive coding framework survives contact with detailed neurophysiological data. Part of the Convergent Science Network podcast series from the BCBT Summer School.

Tagged as:
Apparent Motion Higher Areas Layer-specific Fmri prediction Predictive predictive processing Visual visual cortex

About the author

CSN Podcasts

Both the triumphs of humanity and its most evil deeds have resulted from collaboration. In a time where humanity is required to aspire to the former and minimize the latter, the question arises of how collaboration arises and why it fails. Surprisingly, this phenomenon, so central to who we are, is not well understood. Hence, a collaborative effort is required to understand collaboration in its full biological, psychological, sociological, cultural, and economic complexity and to translate this understanding into operational impact. This series of podcasts is one step toward achieving these complementary goals. The Collaboration Podcast presents interviews with people who are central orchestrators of collaboration in various domains including business, government, science, art, health, sustainability, and the military. The discussions were conducted by Prof. Dr. Paul F.M.J. Verschure and members of the Program Advisory Committee of the Ernst Strungmann Forum on Collaboration (https://www.esforum.de/forums/ESF32_Collaboration.html) during 2021 and had the goal to sketch a map of opportunities, challenges, and obstacles in human collaboration. The forum took place in May 2022, and now we would like to share this series of interviews with a broader audience. The full report of the Forum will be published in 2023 by MIT Press. The podcast was produced by the Convergent Science Network (https://www.convergentsciencenetwork.org/). Context: The stability of social systems depends critically on realizing sustainable methods of “collaboration,” yet how and by which means collaboration is achieved is not clearly understood; neither are the conditions or processes that lead to its breakdown or failure. Collaboration can be understood as cooperation between agents toward mutually constructed goals. Part of the reason for our lack of understanding is that the phenomenon of collaboration is, by nature, a highly multidisciplinary problem, and effective research into its complexities has been difficult to achieve across the broad range of scientific and technical disciplines involved. The need for a fundamental understanding of collaboration, however, has become increasingly important. Not only does humankind demand answers as it attempts to address critical challenges at multiple scales (e.g., climate change, migration, enhanced automation, social and economic inequality), but ever-increasing technological and economic means of interconnecting people and societies are disrupting long-established, familiar patterns of how we interact. Radical technological changes that are ongoing have the potential to reshape collaboration in ways that are currently hard to predict or influence (e.g., by altering configurations in interaction, information creation, and modes of communication). On one hand, such changes could disrupt hitherto stable forms of collaboration by affecting critical communication channels and traditional roles, as can be observed in the rapidly changing patterns in governance, commerce, and social interaction. Conversely, technology could lead to the emergence of novel, successful forms of collaboration that deviate from traditional “hierarchical” architectures. Evidence of this can be seen in areas as diverse as highly automated manufacturing plants, the open science movement, collaborative software repositories, user-centered services, and the sharing of economy-based modes of organization. Without a fundamental understanding of the mechanisms, processes, and boundary conditions of collaboration, it is not possible to evaluate or predict which of these possible scenarios are sustainable or even plausible. The Forum “How Collaboration Arises and Why it Fails” (May 8–13, 2022, Location: Frankfurt am Main, Germany) Chairs: Andreas Roepstorff and Paul Verschure Program Advisory Committee: Jenna Bednar, Julia R. Lupp, Bhavani R. Rao , Andreas Roepstorff, Ferdinand von Siemens, and Paul Verschure

Timestamp

00:00:03 - This is the Convergent Science Network podcast. Leading researchers in the domain
00:00:10 - of neuroscience, brain theory and technology are interviewed by Paul Verschoor and Tony Prescott.
00:00:21 - Paul Verschoor with the Convergent Science Network podcast here at our Barcelona
00:00:27 - Cognition Brain Technology Summer School 2018, together with my colleague Tony Crescott.
00:00:34 - And we have Lars Mugli here. And Lars, welcome to the podcast.
00:00:41 - So Lars, you spoke about internal models and counterfactual cognition predicting
00:00:47 - our environment. And as the title might suggest, prediction was very much at
00:00:54 - the center of your presentation.
00:00:57 - So why do you believe that prediction gives us sort of this strategic lever
00:01:02 - to understand how the brain works?
00:01:06 - I think a few years back, I kind of tried to conceptualize everything I know
00:01:12 - so far from visual processing, let's say 10 years ago.
00:01:16 - And I thought that at some point we have a generally accepted narrative of a
00:01:25 - visual hierarchy that goes of increasing features represented,
00:01:31 - for example, in the visual system.
00:01:34 - And in this story, I felt that there is a particular capability missing.
00:01:41 - Missing um if let's say
00:01:45 - we are driving around in a car and um
00:01:48 - we this would be
00:01:52 - for our brains in this conventional narrative
00:01:55 - kind of an overdrive there's lots of
00:01:58 - varying stimuli there's lots of um
00:02:02 - signal that needs to be attended to that
00:02:05 - are competing for attention it is integrated
00:02:08 - bind bound together these complex features and yet it is possible to effortless
00:02:15 - navigate around and have thoughts about your holiday about your next research
00:02:25 - grant about complicated um.
00:02:30 - Conceptual theories and thoughts
00:02:33 - and so i thought if we use
00:02:37 - the kind of narrative that is coming out of describing early visual cortex and
00:02:43 - then higher stages of visual cortex as the template of how the entire brain
00:02:48 - works we have a very busy and buzzing neuronal system which is combining features.
00:02:55 - Um and we run out of brain space before
00:02:59 - we have an explanation of those really exciting
00:03:02 - things that we are doing so i thought something isn't quite right and um i think
00:03:10 - predictive processing framework brings to the table a narrative that is refreshingly
00:03:16 - different by By saying, well.
00:03:21 - It makes sense if you do something like driving in the car,
00:03:24 - where lots of those features, even though they are quite complex,
00:03:28 - and even though they have spatial temporal dynamics that from a bottom-up system is complicated,
00:03:35 - complicated um you can see that a hierarchical system
00:03:38 - is well equipped to explain that away to to
00:03:41 - say to focus on only the surprising light
00:03:45 - that comes up and signals that you're out of
00:03:48 - petrol and ignore the commercials
00:03:52 - that are running by and these kind of things because lots of
00:03:55 - that can be explained away by different um
00:04:00 - precision explanations you you're
00:04:03 - expecting something to fly by you're expecting commercials
00:04:06 - to try to attract your attention and
00:04:09 - you ignore it so so i think there's a lot that
00:04:12 - can be brought to the table by this top by this global narrative so i mean that
00:04:18 - feels also a bit an idea of how do you let's say optimize information processing
00:04:23 - by actually just looking at your errors so that means stuff you didn't expect
00:04:27 - Yeah, I suppose the stuff you expect. Yeah.
00:04:30 - But was there also something in the data that you were working with at the time
00:04:36 - that was suggestive of moving in that direction?
00:04:39 - Was there sort of a parsimony in the data that you felt needed to be explained in those terms? Yeah.
00:04:45 - So 10 years ago, we have worked in principle with very simple stimuli that looked at a pair in motion,
00:04:56 - which is from one perspective, maybe the most simple visual illusion you can
00:05:01 - think of because it only requires two dots that blink in a certain spatial temporal
00:05:09 - pattern so that your visual system combines them to the illusion of a continuous illusion.
00:05:14 - Movement from this one location to the other location and if it's long distance,
00:05:23 - it may appear as one jump but it's still one object and.
00:05:29 - Commercials work with this, you see this all around, it's everywhere.
00:05:35 - But it's a very simple illusion that we looked at and that we used to show how
00:05:44 - the visual system integrates this information and feeds back to early visual cortex.
00:05:49 - So that was an early finding. finding and
00:05:53 - let's say I started with a surprising finding of some feedback in the intermediate
00:06:02 - space between these dots which have which induce their prayer motion but which is then filled by,
00:06:10 - some activity and we try to make sense of this activity so it was actually driven
00:06:15 - by an experimental result from fmi that this intermediate space was filled up
00:06:21 - with activity that we then thought of how to explain that which then brought
00:06:26 - us to several competing hypotheses one was.
00:06:31 - Related to consciousness and representing
00:06:34 - of an item and the other was more
00:06:37 - related to predictive coding framework which we
00:06:40 - then followed up with several experiments which we
00:06:44 - then came up with a sampling strategy to test
00:06:47 - predictive processing which led us to the conclusion that there is a predictive
00:06:55 - model that is projected down to early visual cortex and and that's how we started
00:07:01 - and the the story i told you about you know 10 or um.
00:07:08 - 12 years ago or so, was when I then left Frankfurt and went to Glasgow.
00:07:13 - And I was thinking of new grand ideas that I put in grand applications.
00:07:19 - And coming from this very simple stimulus, I wanted to make it more realistic,
00:07:26 - using more realistic complex stimuli and using also more interactive subjects.
00:07:33 - So we created one extrapolation was including eye movements.
00:07:37 - Another one was using complex stimuli and looking for predictive processes with
00:07:44 - these more complex scenes.
00:07:47 - Yeah, so I'm interested in how
00:07:51 - and why you arrived at the idea of using apparent motion as the paradigm.
00:07:56 - As you say, you're motivated by this predictive framework and you began by describing
00:08:04 - some ideas from people like David Mumford and his active blackboard idea that
00:08:10 - there is a sort of representation of what you expect to see in the world in higher brain areas.
00:08:16 - And you also described a study from Hanson et al., which talked about,
00:08:20 - if we're watching snippets from a Hollywood movie, we build up this picture
00:08:25 - in our heads in the higher areas of the visual pathway of the meaning of the story and so on.
00:08:30 - And this feeds back and influences processing in the lower visual areas.
00:08:36 - But the task you choose to focus on,
00:08:39 - apparent motion is of course very well known in psychology from centuries back
00:08:48 - the Gestalt psychologists found this and focused on it and really talked about
00:08:54 - it as a way of thinking about.
00:08:58 - Without perhaps too much theoretical insight into it, but seeing it as such
00:09:02 - a dynamical process that's happening in the brain.
00:09:04 - Not so much necessarily a hierarchical process, though.
00:09:07 - So it is a bit of a leap to go from a gestalt process like apparent motion to
00:09:13 - a hypothesis about hierarchy. Okay.
00:09:17 - Interestingly, I think when we started brain imaging, and that goes back to
00:09:21 - 1996 when I joined Rainer Goebbels'
00:09:24 - lab, But apparent motion and other Gestalt laws were kind of the first start
00:09:29 - of how we wanted to look into the human brain to see where these Gestalt laws are represented.
00:09:38 - And motion was something that we started with.
00:09:42 - Actually, the second experiment was imagery of motion, and we're surprised to
00:09:46 - see that that worked and activated V5.
00:09:49 - And while we did these apparent motion experiments,
00:09:54 - one was about bistable apparent motion and the switches between perceiving the
00:10:01 - motion content and not perceiving the motion content was something that you
00:10:05 - can follow in the activity of V5.
00:10:08 - But then we found something in the data that actually V1 knows something about,
00:10:14 - gets informed about some of those integration processes.
00:10:19 - And it was with Nico Kriegesquarte at that time that we were late at night discussing
00:10:26 - what could be the kind of feedback along the trays in the apparel motion,
00:10:34 - how that could be informative.
00:10:36 - And we came then to David Mumford's kind of seminal papers of the 90s in which
00:10:45 - he motivated this active blackboard theory, not totally.
00:10:52 - And being explicit whether this active blackboard is a subcortical thalamic
00:10:57 - structure or primary visual cortex,
00:10:59 - but the idea that these expert areas kind of converge to their best guesses
00:11:07 - in a scene and scribble this onto a blackboard to kind of negotiate the scene
00:11:14 - and the surprises within the scene and so on was quite attractive.
00:11:19 - And we thought okay how can we test this and um
00:11:22 - and i think that started the the following experiments and the paramotion is
00:11:28 - good in the sense that you can have the inducing stimuli far away from a region
00:11:35 - in the middle where you have the illusion and since the the the early part of my brain imaging,
00:11:42 - I always use retrotopic mapping and.
00:11:47 - The trick that the spatial separation of components can be done very cleanly in fMRI,
00:11:57 - maybe better than in some other methods so in EEG you always have everything
00:12:01 - together, you have a better temporal resolution but the signal can be can be separated so well.
00:12:10 - And in fMRI, one of the advantages, you can do individual maps and then get
00:12:15 - the single signals out of those retrotopic components.
00:12:19 - And I think it's this tool that I wanted to explore and use more and more over
00:12:27 - the years up to our most recent finding where we can see that in more complex scenes,
00:12:33 - this occluded part or a rhizotopic space can show that you have something like
00:12:43 - a mental map or mental hypothesis drawn out.
00:12:52 - I think one of the interesting issues here is,
00:12:58 - and that came out in the talk, was how much of the kinds of constraints that
00:13:05 - the brain is using to do something like a parent motion tax are going to be
00:13:11 - implemented within a level of the hierarchy and how many of them are going to
00:13:14 - be higher levels of the hierarchy.
00:13:17 - So I think most of your talk was about how the upper levels might influence
00:13:24 - the process in the lower levels.
00:13:26 - But you presumably would also recognize that there are some processes happening
00:13:30 - within V1 itself which are going to do things like gestalt properties or really
00:13:36 - encourage things like completion.
00:13:38 - Yeah, I mean, this is a very interesting feature of using a parallel motion.
00:13:43 - Motion, because the motion that we trigger is a very fast one and a long distance one.
00:13:53 - So it's a 12 degrees visual angle, it goes to 60 degrees per second.
00:14:01 - Those are features that we, one, cannot process and high visual areas like V5,
00:14:10 - if it's considered or to be higher,
00:14:13 - but specialized visual access trial areas are specialized to process tuning
00:14:22 - along these motion energies, high speed,
00:14:28 - long distances.
00:14:30 - And so it's a particular good case in which we can look into V1's typical feed-forward features
00:14:39 - and see the addition
00:14:42 - of complex features that aren't
00:14:46 - typical to v1 if you want to say in
00:14:49 - a different way v1 has a certain language to
00:14:52 - speak and um we we kind of test the multi-linguality of of v1 because other
00:15:02 - other areas speak in a different language and the question is Because some of
00:15:07 - the interesting and still not totally resolved questions is,
00:15:11 - are these areas translating 2v1?
00:15:16 - So that is v5 translating its prediction
00:15:19 - and explaining 2v1 in a spatial localized way at a very high speed?
00:15:24 - You expect a dot to see here, here, here, here, and here, and this and this orientation?
00:15:30 - Or is it more of a kind of envelope translation to say there's high energy motion
00:15:40 - in this part of the visual field,
00:15:42 - but your knowledge about the concrete incidence, orientation,
00:15:49 - and so on will combine to a better understanding of the situation.
00:15:54 - And that's, I think, the kind of model that I have in my mind.
00:15:59 - There isn't a full translation. I touched upon this with mapping the precision
00:16:04 - of the filling in in complex scenes.
00:16:08 - There's a line drawings, but the precision is around four degrees.
00:16:15 - V1 has a precision of one degree. So it's more like saying there's a line somewhere
00:16:20 - around here, or there's a motion of that kind of speed somewhere around here.
00:16:24 - And this is what top-down can give as a prediction and combined with the lateral information saying,
00:16:34 - oh, there was a dot that just disappeared here together with the energy prediction up there,
00:16:39 - I can now make a very precise prediction.
00:16:43 - By combining those two constraints.
00:16:46 - But before we get there, because that's sort of after a number of experiments, right?
00:16:51 - Right. And I think that the first question is whether, given the data you have
00:16:57 - on this apparent motion paradigm,
00:16:59 - whether this predictive mind hypothesis or the active inference hypothesis,
00:17:08 - whatever you want to call it,
00:17:09 - is the most parsimonious explanation of the data that you have.
00:17:12 - Because in what you showed in your first experiment, it had apparent motion.
00:17:16 - So we have the two fields are being stimulated in a certain order.
00:17:20 - So you have this idea that something moving.
00:17:23 - And now you're going to flash a stimulus in the center area either synchronized
00:17:28 - with disappearing motion right or out of out of sync out of phase with this apparent motion,
00:17:34 - uh where you you look at the time difference of one frame right so this would
00:17:39 - be about 15 milliseconds or something like this i don't know so it's a very
00:17:42 - slight deviation right of the predicted path.
00:17:45 - And then what you do observe in your fMRI is that eight seconds after,
00:17:55 - stimulation, either congruent or incongruent,
00:17:59 - you see a slight deflection or deviation of the BOLD response in the area that
00:18:06 - you stimulated with this predictor-unpredictor stimulus,
00:18:09 - which is slightly higher the bolt response is slightly higher for the deviant
00:18:15 - stimulus as compared to the congruent stimulus this is the main effect and the deviation,
00:18:21 - the bolt signal is of the order of about what 8 to 10 percent,
00:18:27 - something like this right and it's also slightly enhanced in both cases relative
00:18:33 - to the baseline response,
00:18:35 - so now Now, why do you believe that this predictive coding into the power hypothesis
00:18:42 - interpretation is the most parsimonious explanation of the data?
00:18:50 - Andy Clark has a wonderful cover picture in his book on predictive processing,
00:18:57 - I think, or it's called differently.
00:18:59 - It's Surfing Uncertainty.
00:19:03 - And if you take the picture, we're in Barcelona. I've seen the beach this morning.
00:19:08 - The model is a little bit like this. You have this wave of motion energy, of prediction.
00:19:17 - It is translated to something that needs energy in V1, like a prediction.
00:19:25 - And then you place on this wave a surfer that serves the waves and then is efficient
00:19:31 - and comes through and is detected. That's the dot that's in time.
00:19:37 - And the surfer that has a little bit of a timing problem won't make this and
00:19:42 - will fall off the wave. And that's the prediction error.
00:19:47 - And that makes a small difference. And the surprising result is that this one
00:19:52 - frame difference is detectable in bold signal.
00:19:55 - And my most parsimonious explanation the way I think is,
00:20:00 - yeah, you see at this moment the mismatch between the prediction and the surfer,
00:20:07 - the dot that doesn't quite make it to be within this envelope.
00:20:11 - Now, as I said earlier, this envelope moves very fast, 60 degrees visual angle.
00:20:16 - That's faster than any motion detector on B1.
00:20:20 - B1 has lots of lateral connections, but I doubt that they have the speed to
00:20:25 - process this. But I'm not opposed to this explanation.
00:20:28 - I've had several discussions. People are on both sides saying,
00:20:32 - well, there could be a contribution.
00:20:35 - And in the model that I just pictured before, I do think there is a contribution
00:20:39 - of lateral interaction given the feedback signal of a certain motion envelope.
00:20:46 - And giving the neighborhood relation of the just disappearing dot.
00:20:53 - Gives the high precision of of this well i
00:20:57 - would say the long range long range lateral interactions in v1 are iso orientation
00:21:03 - right so neurons that like similar features linked together over long distances
00:21:09 - and that would of course be a perfect substrate to exploit to get a fair motion because,
00:21:16 - the activity will be very actively guided along neurons with a similar response
00:21:21 - tuning to move in a certain direction, right?
00:21:24 - So this would give you essentially a kind of entrainment response,
00:21:28 - right, to explain the parent motion, which might be also more in line with actually
00:21:33 - the Gestalt ideas, right?
00:21:37 - So can you… For this model, that's
00:21:42 - why I like to use the image of the
00:21:47 - paper of mock
00:21:52 - I'm blanking on her name but and who did their promotion with the gratings and
00:22:00 - the filling in is inducing a new feature which is a new orientation which is
00:22:07 - not just neighborhood relation it's a smooth transition now.
00:22:14 - Maybe the case can be made that the lateral interaction smoothly goes from one
00:22:18 - orientation over space to bleeds into other orientation.
00:22:26 - But it's not such that the feature that's picked up on the paramotor trace is
00:22:32 - just replication of the neighboring feature in orientation.
00:22:37 - So to me, it looks more as a constructed feature.
00:22:43 - But that would just be a matter, as long as I take enough freedom to fiddle
00:22:47 - with the topography of the lateral interactions, I could also get this apparent feature, right?
00:22:53 - But another answer, of course, could be you could still say,
00:22:56 - look, you see, we agree, because the predictive model still holds,
00:23:01 - it's just that the substrate is also included already in V1 wiring.
00:23:05 - But for some reason, in your view, you want to see it like a hierarchical system, right?
00:23:11 - And without ascribing a big functional role to the local structure of V1 in this case.
00:23:18 - Well, you also know that 97, 98% of the synapses in the V1 volume originate inside V1.
00:23:29 - Only a tiny fraction comes from outside V1, right?
00:23:33 - So, do you believe that this tiny fraction of long-range projections coming
00:23:40 - out of V2 or out of your thalamus or other cortical areas are sufficient to
00:23:46 - carry your predictive model?
00:23:49 - So, there are two lines of evidence where we really looked at this interaction.
00:23:53 - One that I presented, one that I didn't present.
00:23:57 - And maybe to add the one that I didn't present, we used an EEG experiment.
00:24:03 - Again, with a prior motion design.
00:24:05 - And we found two components that have energy components.
00:24:13 - Motion energy components as compared to flicker to
00:24:16 - flicker and those components were around 100
00:24:19 - milliseconds and around 140 milliseconds after
00:24:22 - stimulus onset and we then had the question which one is from retinotopic regions
00:24:32 - and which one is from v5 and what we found is we applied the a paramotion in
00:24:38 - the upper visual field and in the lower visual field,
00:24:41 - which induces an EEG component in the dorsal and in the ventral stream,
00:24:46 - so they have different orientation.
00:24:49 - And we subtracted one where we had a paramotion in the upper visual field from
00:24:53 - in the lower visual field and found that the early component was subtracted away,
00:25:01 - meaning that the motion sensitive area V5, which in both cases has the same localization,
00:25:09 - is the early one, around 100 milliseconds, and the later one,
00:25:13 - the 140, is the upper and the lower of the retrotopic.
00:25:16 - So the retrotopic component comes 40 milliseconds later.
00:25:23 - So the other experiment I showed was the TMS experiment at which we stimulate
00:25:29 - V5 50 milliseconds, 40 milliseconds before the onset of the flicker,
00:25:36 - which is on the power motion trace.
00:25:38 - And this TMS takes away the predictability effect on the power motion trace.
00:25:44 - So both of those indicators, I think, speak more to a communication between V5,
00:25:52 - which is tuned to process this high velocity motion across huge space.
00:26:03 - And the more localized features in v1.
00:26:08 - And so therefore I think it's a good paradigm because.
00:26:13 - Different features are processed in different different regions and and they
00:26:18 - are naturally optimized for this and you can see this kind of interaction um
00:26:22 - but now it's an alternative,
00:26:24 - for for your two experiments i could also argue look see your first experiment
00:26:28 - you show v5 leading v1 in this response right and um then i could argue about
00:26:35 - maybe you need a minimum volume of response actually be detectable with your methods of fMRI.
00:26:42 - This kind of, given the heavy convergence to V1, you will only get that initial
00:26:49 - critical response that is detectable at the V5 level and after the V1.
00:26:54 - So it's more an artifact of your measurement technique than really reflecting
00:26:59 - the underlying dynamics. Can you exclude that?
00:27:02 - So it's true that we usually have these block designs, but in the EEG experiment, we measure these.
00:27:16 - Short, I mean there's 100 milliseconds after onset of the apparent motion energy
00:27:22 - component as compared to blinking for example and also in our experiment where
00:27:28 - we combine apparent motion with saccades it's such that it even works after
00:27:33 - the saccade in the new hemisphere.
00:27:36 - Which Which, if I understand you correctly, you're saying you need repetition
00:27:42 - over time to build up the V5 signal.
00:27:46 - That's convergence. That's plain convergence.
00:27:49 - That you just have a sufficient drive onto that volume of cells to become really
00:27:54 - a clean signal that you can extract with your method.
00:27:58 - As opposed to a more distributed, diffused signal in V1 that is more difficult to detect.
00:28:03 - Well yeah i mean if you like in if if you take v1 after the saccade a blinking dot,
00:28:12 - there's the inducer the last stimulus of their paramotion
00:28:15 - and a blinking dot next to it the target those two are always perfect a paramotion
00:28:22 - if that would be the only event if v1 in the new hemisphere would process that
00:28:27 - on its own they are as much related to one another the in time is as much related as the out-of-time.
00:28:34 - The only thing that makes these two stimulation conditions different is the
00:28:41 - history that was processed in V5 and in V1 in the other hemisphere.
00:28:46 - So it's bringing this history to this new situation that creates a difference.
00:28:58 - It might be true that it needs to build up over time, this hypothesis in V5,
00:29:02 - because a prior motion is a peculiar situation.
00:29:06 - If you have just two dots and you perceive them as a prior motion,
00:29:09 - there's no other way describing that as postdiction.
00:29:13 - You need to, after the second event only, you can understand that it was a motion
00:29:19 - and every point in between would be integrated at a later time point.
00:29:25 - So it's like you're reversing the time frame.
00:29:31 - In our experiments, we have always had eight iterations of a pair motion.
00:29:35 - So that takes care, the visual system kind of has time to catch up with the
00:29:46 - delay in simulation to then run in time.
00:29:50 - Time if the famous experiment is if you're doing a pair of motion you take away one stimulus,
00:29:56 - so it's repetitive and then you take a
00:29:59 - stimulus away you will still continue to see a pair of motion because your visual
00:30:02 - system is now so predictive that it fills in this that it can catch up it needs
00:30:07 - to do post-diction of the missing stimulus to to realize so your fair motion
00:30:12 - is there but the question is do we really see a strong contribution to the phenomenon
00:30:18 - at the signal level of something we might want to call a predictive model, right?
00:30:24 - So then also for your second example, you could also see where you talk about your TMS experiment.
00:30:30 - So look, if I zap V5, I can sort of disrupt the error detection signal.
00:30:38 - This is how you would interpret it, right? And then the error detection signal,
00:30:42 - the deviant trigger signal signal ends up roughly the same magnitude as the congruent stimulus.
00:30:50 - But that's, of course, I could still
00:30:52 - argue, but that's at best a necessary condition of a predictive model.
00:30:55 - And it doesn't prove in any way that what comes from V5 is a predictive model
00:31:01 - that is processed into an error at the V1 level. Can you exclude that?
00:31:13 - I'm struggling to find a way in which we could prove that. your distinction how,
00:31:23 - so which aspect are you,
00:31:27 - questioning the content of the model so to say V5 speaks to V1 and okay,
00:31:36 - and the content of
00:31:40 - the message has something to do with the power motion and And are you now saying
00:31:48 - there could also be a contribution of lateral interaction in V1 that contributes
00:31:52 - to creation of this complex prediction that we test?
00:31:58 - Right. So this is an alternative explanation that so far you couldn't exclude yet.
00:32:03 - No, I'm not. Except that you're saying, well, but I have a TMS experiment.
00:32:07 - And then I'm saying, yeah, but wait, I could still have all this process playing
00:32:11 - out at the V1 level. We know anatomically V5 projects back to V1.
00:32:16 - So if I zap V5, it's not so strange. Something happens to the V1 level,
00:32:20 - but it doesn't tell us anything about whether this is a prediction being fed
00:32:24 - back or just there's an anatomical connection between the two areas and signal exchange.
00:32:31 - So I think Jeff Hawkins uses this kind of explanation of this hierarchical predictive
00:32:36 - memory prediction framework, which is as you know, you...
00:32:42 - At every level, the neuronal microunit
00:32:48 - is trying to explain its own activity given the surrounding stimulus history
00:32:56 - in the context of a message from a higher level top-down prediction.
00:33:05 - So and the example
00:33:08 - he's using is like if you have a melody then
00:33:13 - the higher up areas in
00:33:17 - auditory cortex would tell you what is expected to continue in this melody and
00:33:23 - giving the local information of transients at this lower level and giving the
00:33:30 - envelope context from higher you create these neurons kind of create what's
00:33:35 - most likely to happen next.
00:33:37 - And so it's within this kind of model that I would also, and there's actually
00:33:41 - a good model that does that for a pair of motion or for motion behind occluders,
00:33:48 - where it does exactly this, that the top-down is the kind of envelope and the
00:33:53 - lateral is the additional information that then converges to a precise prediction.
00:33:58 - So, the example here being you have a motion of a certain energy disappearing behind a.
00:34:10 - Occluder, and the way it's modeled is there's a motion energy expected in this area, and.
00:34:20 - But you can't be very precise where it is.
00:34:24 - The disappearance of the dot gives an additional constraint.
00:34:28 - And these two information together make a very precise prediction of the trajectory
00:34:36 - of the dot. And that's the way I think about it.
00:34:38 - So I'm not excluding the lateral interaction. So you're saying then that the
00:34:44 - idea of a hierarchical predictive model is a functional concept that's not mimicked
00:34:50 - or in sort of an isotropic way mapped to the anatomical hierarchy,
00:34:55 - because it might be implemented by some classic confluence of recurrent projections
00:35:00 - and local interactions in neural circuits. Exactly. This will be your point. Yes.
00:35:07 - All clear. because also in that sense you
00:35:10 - made the point and also resonates with
00:35:13 - this the traditional view and that's also where
00:35:15 - Hawkins I think sits would be rather dogmatic in that sense right there's a
00:35:20 - top down prediction and this comes together at a lower level in the hierarchy
00:35:26 - where it leads to an error there's a real comparison taking place between the
00:35:30 - state that this lower level area believes is correct
00:35:34 - it gets a reference as you want from a top-down area and now a comparison happens
00:35:38 - and I have an error right and your data would not really reflect that in such a literal sense right,
00:35:46 - because you in in essentially in your what you'll see a ball signal for the
00:35:51 - matching the congruent stimulus also leads to a deflection of the signal as
00:35:55 - does the incongruent stimulus although the deflection is somewhat different
00:35:59 - right so that means it's not about the error processing So what do you then
00:36:04 - think is being processed,
00:36:05 - is being generated in that local V1 circuit if it's not an error as the traditional
00:36:12 - model suite would predict?
00:36:15 - Well, I think this is, of course, a very good point, but it seems like a point
00:36:22 - about the labels we add to those coding differences, right?
00:36:28 - So if a prediction is violated and the violation of the prediction gives a different signal,
00:36:38 - or whether it's a confirmation of a prediction that gives another signal,
00:36:43 - It is the combination of input signal and expectations that are combined.
00:36:53 - And they can be subtracted, which is the prediction of the predictive coding proper way,
00:37:04 - or it could be multiplied and resonate.
00:37:08 - And it's very difficult to define the level at which we can resolve that debate
00:37:16 - because I'm currently not sure how we would resolve this because at the neuronal
00:37:25 - level, so what is the right level of description?
00:37:28 - We have the microcircuitry within V1 that is a combination of excitatory and
00:37:37 - inhibitory neurons neurons,
00:37:39 - the error units that can be also a combination of excited or inhibitory neurons,
00:37:47 - so it will be very difficult to resolve that.
00:37:50 - But the narrative of the predictive processing framework is very clear about
00:37:56 - those units in principle. So prediction error matters.
00:38:01 - Reduction of prediction error or...
00:38:07 - Is a currency that might be very useful.
00:38:12 - And also in the predictive coding framework, you have the resonating,
00:38:18 - the confirmation signal for a predicted signal that keeps an internal model running.
00:38:26 - So you have all those components, And I think it's, you know.
00:38:36 - It's going to be, do you have a suggestion how to solve that?
00:38:40 - I mean, do you have, is there some variance that is not explained or that would
00:38:45 - be more possible to explain by an alternative explanation?
00:38:48 - I'm not so sure. Yes. Okay.
00:38:51 - Temporal populating codes. But before we get there,
00:38:54 - so I feel one way, if I would be a real cynic and observing your results,
00:39:02 - I could say, look, you have done a fabulous job dismantling the predictive modeling framework.
00:39:07 - Work, because the second thing is, also if you go back to the traditional idea
00:39:11 - of, you mentioned Mumford,
00:39:13 - and then we have a number of other people who have variations on that,
00:39:17 - and before that also Barlow was talking about it, you would expect that in the
00:39:22 - cortical circuit, and it was very literally mapped to the anatomy,
00:39:25 - that in a cortical, people would put several layers in the cortex that would
00:39:30 - then deal with the prediction and with the current state and how the error would be computed.
00:39:35 - So very precisely to cortical circles, right?
00:39:38 - But even if you do your layer-specific analysis, you don't necessarily see such
00:39:44 - an asymmetry among the layers in their response.
00:39:49 - There seems to be a very gradual distribution of layers.
00:39:54 - Of the signal that you get in your task. In that case, it was more like an occlusion
00:39:59 - task, but I think it makes the same point.
00:40:01 - So in that sense, aren't you showing it actually goes on in these circuits,
00:40:06 - might reflect aspects of prediction, but it's not necessarily playing out in
00:40:11 - the way as anticipated in these more traditional models.
00:40:14 - Would you agree with that? I'm totally experimental and not dogmatic, right?
00:40:21 - So these are interesting questions.
00:40:24 - I want to see how does traditional neuroscience looks at the stimulus response
00:40:33 - to an unpredicted random stimulus.
00:40:36 - And of course, in those experiments, you can never see the capability of predicted sequences and so on.
00:40:47 - So you need to do the experiments in which the stimulus history is predictive
00:40:53 - for a certain variation and so on.
00:40:57 - And I think that's what we have done and others and whether this creates a,
00:41:06 - I mean, the one point in which I deviate with the Rauer and Ballard model, if you like,
00:41:13 - is that what we see is that the top-down prediction is creating a signal in
00:41:21 - where there is nothing, right?
00:41:23 - So predictive coding is a story in which you explain away as much data as possible,
00:41:31 - but now we show that in the non-stimulated region, where there's nothing to
00:41:36 - explain away, there's something created.
00:41:39 - There's a model, there's a prediction, a created and used energy to put their
00:41:45 - hypothesis to then be tested.
00:41:47 - It um so in in this respect
00:41:50 - it deviates and um and i mentioned flores
00:41:53 - de lange showing similar results for illusory contours they
00:41:57 - are created they are put into into a map it's an active blackboard framework
00:42:02 - in which you know the chalk is picked up and to to draw something on the active
00:42:07 - blackboard and say i you know it would make sense if there's something moving
00:42:10 - and it would make sense if there's some information missing at these and these points.
00:42:15 - In a global sense, that might be very useful and minimize energy in this sense
00:42:22 - that it prepares the organism to respond to more or less surprising stimuli.
00:42:30 - But it's not all about a quick explaining way of all energy in hierarchy as
00:42:41 - it was shown by Rau and Ballard,
00:42:43 - which is just a very simplified model to illustrate some principles of predictive processing.
00:42:54 - Um, yeah, so combining with that, also what we, you also mentioned much more
00:43:01 - detailed physiology that you did with Larkham and other people inside the O-Brain project.
00:43:06 - And also there, you went over those results rather quickly,
00:43:10 - but still at best they showed there is indeed interaction between higher and lower areas,
00:43:16 - but it was not necessarily a clean signature of anything that might only interpret
00:43:21 - like a strong prediction or or an error, or anything along these lines.
00:43:26 - It seems a more non-specific sensory processing that depends on the active dendrite.
00:43:32 - Yeah, so, I mean, this is ongoing data that is recorded and still analyzed.
00:43:42 - Like, well, I think in the classical V1 IceCube model and following some claims that, you know.
00:43:55 - You know, Jack Gallant would say, you know, it's almost explained,
00:44:02 - you know, 80% of the variance is explained in V1.
00:44:05 - We have a very good idea of our models about V1 and that is true actually for
00:44:11 - in certain restrictions.
00:44:15 - So it is in those models in which you have a strain of surprising stimuli,
00:44:21 - it doesn't explain ongoing activity,
00:44:24 - baseline activity, and these kinds of things, which are a huge contributor to
00:44:28 - the energy level that is boiled off in B1 volume.
00:44:36 - So it's maybe only the waves on top of that that are then explained.
00:44:40 - Now, in a classical receptive field V1 explanation model, you don't have the
00:44:48 - description of these feedback signals that I described.
00:44:55 - Another one that I haven't described today, but we have done research is in
00:45:00 - blindfolded subjects when we play auditory scenes, you have some activity in
00:45:05 - V1 that is related to these auditory scenes.
00:45:10 - Um, so the, the, the, the properties and the mechanisms in the envy one.
00:45:18 - Are still having room for negotiations
00:45:23 - of expectations and predictions in whatever kind of mechanisms.
00:45:28 - And I think Christian Liefeld is doing a wonderful job in decoding also the
00:45:35 - level of interneuronal activity that contributes to the detection and the explaining away.
00:45:42 - And I think he has found a subtype of
00:45:46 - inhibitor neurons that is very strong in exactly being silent during visual
00:45:57 - stimulation and being active in between stages.
00:46:03 - So it could be a role of working hard to explain things away that we still will find out.
00:46:12 - You know, um, so there are many sub units that can still be,
00:46:16 - um, contributing to this, to this model and to this compartment of visual cortex
00:46:22 - that we think we, we understand best.
00:46:25 - Maybe we want this thought to be one of those regions that are best studied and best understood.
00:46:30 - And yet there's, there are like languages of feedback signals that we don't fully conceptualize.
00:46:39 - You mentioned one of the results that you found was a difference between paranoid
00:46:45 - schizophrenic patients and controls on this apparent motion task.
00:46:50 - So what do you read from that in terms of support for your hypothesis about top-down predictions?
00:46:57 - And also, what does it tell us about schizophrenia?
00:47:00 - So we actually didn't find a difference between schizophrenic subjects and controls.
00:47:09 - Not in the dimension that we expected it, in the predictability effect,
00:47:13 - because we could confirm the predictability effect.
00:47:16 - We saw overall a difference in the amount of a paramotion perception.
00:47:22 - If you like, the responses to detection of a paramotion was different in controls in schizophrenia.
00:47:34 - I mean, there's a general result, isn't there, that schizophrenics are less
00:47:37 - susceptible to some visual illusions?
00:47:39 - Exactly, yeah, like the hollow face mask illusion, and the same is true for paramosion.
00:47:46 - But what we didn't find is that the effect that we see,
00:47:51 - the advantage of a flickering stimulus that is consistent within a paramotor
00:47:56 - context was more or less in schizophrenic subjects than in control subjects.
00:48:05 - So the original idea was that it could be that the signatures of creating a
00:48:17 - prediction error and the processing of a prediction error are altered in schizophrenic subjects.
00:48:21 - Subjects, they are less tuned to process the prediction error,
00:48:25 - or higher tuned. Actually they are both hypotheses.
00:48:29 - But we didn't find that, so we basically just replicated this.
00:48:34 - Now we are testing the same thing in autistic subjects, we're just starting
00:48:38 - to do that, same experiment.
00:48:43 - Some have suggested that it should precisely be the difference between autistic
00:48:48 - subjects and schizophrenic subjects, in which you have this low-level difference
00:48:53 - in visual predictability,
00:48:56 - that matters in autistic subjects but not in schizophrenic.
00:49:01 - So we will see that. But the prediction here would be that autistic pupils would
00:49:10 - be less tuned in explaining away the predicted stimulus.
00:49:15 - It would be as unexplained, as surprising as the unpredicted one.
00:49:23 - And, well, it shows a tuning for predictability that is important and essential.
00:49:29 - And we could think of it in the sense that if you experience lucid dreams,
00:49:40 - it's maybe a situation you're aware, you're conscious, but you don't respond to prediction errors.
00:49:51 - Uh you the the story you don't wake up and realize you are in your bed but uh
00:49:59 - you continue your dream and knowing being in your bed and knowing to dream does not um.
00:50:07 - Bring the dream to an end and and so that's that's maybe a situation that that
00:50:12 - comes very closely to to a auditory hallucination in in schizophrenic subject
00:50:18 - it is it is causing some prediction error They're surprised,
00:50:21 - they're worried, but it's not resolved.
00:50:24 - It doesn't make the internal representation
00:50:29 - go away or being replaced and overwritten by an alternative.
00:50:34 - SL. But now in terms of the actual data, in a schizophrenics issue,
00:50:39 - we do the parent motion task.
00:50:40 - How big is the deflection you would see in their birth response as compared to a healthy control?
00:50:47 - What is the exact difference? Uh, so, um, sorry for having been so brief,
00:50:53 - but in the, in the talk, I only presented behavioral results and we haven't
00:50:56 - done functional brain imaging with him.
00:50:58 - It's just a behavioral observation that they are not detecting the.
00:51:05 - In-time stimulus as good as the... What would be your prediction in much respect
00:51:10 - to what you would see if you would do fMRI on them?
00:51:13 - Yeah, that's a good one, right? In the autistic subjects, if we find behavioral.
00:51:20 - No difference between the predicted and the non-predicted flash stimulus,
00:51:26 - we would like to do the fMRI experiment using layer-specific fMRI to see whether
00:51:32 - then we have a continuous low-precision prediction in the superficial layer
00:51:37 - that doesn't discriminate and doesn't trigger an error signal could be one hypothesis.
00:51:44 - So another one would be that a prior motion is never creating any activity along
00:51:52 - the trace, so that the envelope is maybe processed feed-forwardly,
00:51:57 - but there's no feedback message.
00:51:59 - So everything that comes in hits the clear slate.
00:52:05 - So now, since I'm on this crusade to sort of demolish the predictive model,
00:52:11 - another piece of the data that feeds that assault is the data that you presented
00:52:19 - on the CIRBAN, which is really interesting, right?
00:52:21 - Because when you opened up your analysis in your whole brain,
00:52:25 - you also look at how other structures would be involved in this.
00:52:28 - Now, soon the cerebellum starts to also show activity in this task, right?
00:52:33 - So now I could say, aha, you see,
00:52:38 - apparently this sort of clean isomorphic mapping of prediction hierarchies to
00:52:42 - cortex is insufficient to explain the paradigm, to explain the behavioral effect, right?
00:52:51 - So how does the results you get from the cerebellum not question this predictive
00:52:58 - hierarchy model that we started out with?
00:53:02 - So I had been a cortical chauvinist by accident, not by conviction.
00:53:09 - So we just started out scanning cortex because cerebellum didn't make it into
00:53:17 - our slab. We have changed that now.
00:53:20 - And I'm still on my learning curve to learn more about the cerebellum.
00:53:26 - How much I understand of it, or some of the hypotheses are that it is a machinery
00:53:34 - that has an architecture to create a temporal forward model that is used in,
00:53:45 - for example, in tracing behavior.
00:53:49 - So seeing, you know, tracing a curve.
00:53:52 - Monkeys can follow this curve by smoothly,
00:54:00 - which is a visual motor integration task,
00:54:04 - that by taking away a cerebellum, this movement becomes very jittered and incoherent and so on.
00:54:16 - And there's a long line of different research, but so the cerebellum in a way
00:54:22 - is not directly connected to the outside world.
00:54:27 - It receives cortical input, processes this, and is the archetypical region for
00:54:35 - an afferent copy processing, which is basically a predictive machinery.
00:54:41 - And so, as I've heard in a recent talk, if evolution came out with a perfect
00:54:48 - prediction machinery and is connected potentially to the entire brain.
00:54:55 - What would you use that machinery for?
00:54:58 - And so I think the conclusion is that lots of the forward modeling.
00:55:08 - That is necessary for survival is done,
00:55:14 - is offloaded, or is done under the participation with the cerebellum.
00:55:18 - So it is, if you like, our predictive machinery that communicates then with
00:55:25 - the different sensory and motor output areas.
00:55:31 - Now, this was a funny one.
00:55:33 - The Cervelo projections are mainly going to frontal parts of the cortex, almost densely.
00:55:39 - So then to get that signal back into V1, we can take some cascade of recurrent projections.
00:55:47 - I thought so. that there are some I think there are some direct connections
00:55:53 - to visuals so there's some visual.
00:55:57 - Cerebellum connections I'm not entirely sure if they go to V1 or to exostride
00:56:01 - areas and so on but I think there must be something like a visual sub-part of
00:56:07 - cerebellum Sure Okay so the last part of your talk talk,
00:56:16 - went a little bit more in the direction of a more computational understanding
00:56:21 - of what this whole system might be doing, right?
00:56:26 - So the bottom line would be Cortex builds hierarchies of internal models that
00:56:36 - are coupled through prediction errors.
00:56:40 - Um and so that's
00:56:43 - the starting that was a starting point and that's been a great heuristic for
00:56:47 - you to to perform some some really fantastic experiments that
00:56:50 - also led to to new insights but new insights would
00:56:53 - also allow us to to reformulate the model right so so if you would have to sort
00:56:59 - of crystallize that that current view what would be your your current summary
00:57:02 - of of the model or the theory of of neocortex how should you think about neocortex
00:57:09 - given the data that you have in your hands?
00:57:15 - I think that, of course, one of the big struggles is to find the right level
00:57:25 - of abstraction in answering your question in the description,
00:57:28 - right? How many units do we need?
00:57:30 - What kind of abstraction is useful for what kind of understanding?
00:57:34 - Understanding we're still on we neuroscience is a data rich um science with
00:57:41 - relatively little theories um at least that's what some of the criticism um
00:57:47 - says about neuroscience so um,
00:57:51 - and we haven't this is a question we haven't fully resolved which level of description
00:57:56 - do Do we need spiking neurons explanation? Do we need principles?
00:58:02 - And I think this predictive processing has some hypothesis,
00:58:10 - some units, which could be in single neurons or microcircuits,
00:58:16 - which is the description of prediction errors, for example, and important currency, I think.
00:58:21 - And also the kind of um.
00:58:28 - Population code of internal models so
00:58:32 - some for more complex uh human
00:58:35 - tasks it is a it is extremely important to be
00:58:38 - able to simulate an counterfactual situation
00:58:42 - to plan behavior to plan a interaction
00:58:46 - or to plan a career we can think about
00:58:49 - i mean there are many very fascinating explain you
00:58:53 - know how how can how can you explain that a
00:58:57 - biological system is capable of committing suicide how how did evolution make
00:59:03 - it possible that you can you know you create an internal model that is somehow
00:59:09 - rewarding that you simulate and say that makes sense and then commit suicide which
00:59:14 - in biology is extremely unbiological,
00:59:19 - right?
00:59:20 - So there is only a few components that you need to be able to explain how internal
00:59:31 - models can create a good description about external facts, right?
00:59:37 - So deep encoding works well in the visual system, for example,
00:59:40 - to now label visual scenes and learn the different objects by coming up with
00:59:49 - a condensed description,
00:59:54 - but just by learning an incredible amount of visual stimuli and trying to extract
01:00:01 - the most essential features.
01:00:04 - So I think there are some of those components,
01:00:08 - that are sufficient to come up with a hierarchy of extracted features by using
01:00:19 - prediction error minimization.
01:00:25 - But I think what's unresolved so far, what I want to look into future,
01:00:29 - if that was your question, is the kind of, how do you lift up this internal
01:00:36 - models that are totally, that are then totally connected and well describing
01:00:40 - the environment to something that is alternative,
01:00:43 - that you can think, plan alternative to the currently existing outside world.
01:00:52 - I was looking for an alternative to the classical model.
01:00:55 - I don't really hear it yet, but Tony. Well, you mentioned the deep convolution
01:01:01 - neural nets a couple of times in your talk and just now.
01:01:04 - And, you know, sort of Paul and I are both interested in putting models of the
01:01:09 - brain into robots and getting them to do tasks in real time.
01:01:13 - So, and I take from some of the things you're saying that maybe there's some
01:01:18 - useful mileage to be had in using these convolution neural networks works as
01:01:22 - an approximation to what the visual pathways might be doing.
01:01:27 - But, I mean, say we get that working in some sort of first pass,
01:01:32 - what would be the things that we would be missing out on and what would we want
01:01:37 - to add to make this a more realistic model?
01:01:39 - So I think the flexibility, the
01:01:45 - adaptation to new environments is something that is a potential of expanding
01:01:56 - deep encoding neural networks that are feed-forward networks by expanding to recurrent looping.
01:02:05 - Might be able to make them even more clever.
01:02:11 - To, let's say, the context-dependent amplification and de-amplification is something
01:02:21 - that could potentially,
01:02:24 - once we find the right architecture, make deep encoding networks more flexible
01:02:32 - to adapt to new environments.
01:02:34 - So this is, I mean, you're more
01:02:37 - the experts in this, and I'm sure this is something that's widely debated.
01:02:42 - From our research, we just see the great difference between.
01:02:48 - The big success story of deep encoding networks, convolutional networks,
01:02:54 - that are very similar to the hierarchy in the visual system.
01:03:00 - However, they're both feed-forward.
01:03:02 - So they're feed-forward processing architecture.
01:03:07 - And the only thing that's fed back is the error signal.
01:03:14 - But if we take our research serious and add a feedback cascading network to the feedforward,
01:03:30 - then what we get, for example, is a network that can fill in occluded images,
01:03:35 - which is one of the examples that we have used in our lab, in which we see that
01:03:41 - humans have line drawings fitted in by their visual cortex and our deep encoding networks,
01:03:50 - connected to an autoencoder, so a U-shaped network, is able to then make predictions
01:03:57 - about occluded objects.
01:04:02 - Images that are currently not in sight.
01:04:05 - And so that could be an architecture that is also.
01:04:14 - If you like, an active blackboard of predictions, what is happening when something
01:04:19 - disappears over time and makes the world enriches, so to say,
01:04:25 - the environment in which the network is working.
01:04:28 - But wouldn't that imply that we would need much more brain volume with our skull
01:04:33 - and the size of a Skippy ball because for those models, if you want to bring
01:04:39 - in any kind of invariance, scale, rotation, position,
01:04:42 - you have to duplicate your wires, right?
01:04:45 - So you would run out of wires very quickly. So would it scale?
01:04:48 - Would that approach really scale, you think? I mean what we have done for the
01:04:55 - U-shape network is just doubling the network right it's a,
01:04:59 - so after the conversion it diverges and has lateral connections and to reconstruct
01:05:06 - the images and that makes it to if you like an active blackboard architecture.
01:05:17 - Nico has done it in a slightly
01:05:21 - different way used object recognition task
01:05:24 - in a network that is either feed
01:05:27 - forward or has also lateral and
01:05:30 - feedback connections this blt network and could
01:05:33 - show that this kind of network architecture um
01:05:37 - is better in recognizing overlaid numbers cluttered scenes and so on and that
01:05:46 - architecture came from kind of joint discussions we had about you know the question
01:05:51 - how can we how would you add what would feedback you know,
01:05:58 - add to your feedforward networks? How could it improve that?
01:06:02 - How could we design experiments that challenge the current recognition system?
01:06:06 - And so cluttered scenes and overlaps was one of those examples that he tried.
01:06:12 - And I think, you know, the Alex Neck kind of architecture had something like
01:06:19 - 88% correct or more, or 95% correct classification.
01:06:24 - And then his BLT, bottom-up, lateral, and top-down network,
01:06:33 - was a few percentage better or part of percentage or something significantly.
01:06:40 - So it's small, but you need to find new tasks and new challenges that those
01:06:46 - networks can play out and maybe their potential because they're already in object recognition.
01:06:52 - That seems to be a field where they are ceiling now.
01:06:59 - So coming back to the point, it might be that they are now more flexible,
01:07:03 - maybe there are advantages in training rates or something. I'm not an expert.
01:07:10 - So we need to add in these sort of top-down cascades and we can get some improvement
01:07:15 - and presumably some sharpening of the representations in B1 I think one of the
01:07:23 - times you said in your talk,
01:07:24 - you know, you usually think of V1 as the bottom of the hierarchy,
01:07:27 - but in a sense, it's much higher up because the information ascends and comes back down.
01:07:33 - So you're reconstructing this high resolution version of the scene. But I mean, the...
01:07:41 - You also talked about how we can apply machine learning algorithms or support
01:07:48 - vector machines to read out from the brain to what the brain is actually seeing.
01:07:52 - So it seems to me that in some ways we're getting back to what really was an
01:07:59 - old idea about the visual system,
01:08:01 - that the brain sort of, or the visual brain represents its best
01:08:07 - guess at what's in the world in some almost literal sense.
01:08:11 - The whole idea of the movie screen in the head, which was dismissed by everybody as naive.
01:08:18 - But with these kind of approaches, with this notion that you're getting this
01:08:22 - high-resolution map which is tuned by your predictions,
01:08:27 - is there in some sense something like a representation in V1 of your best guess
01:08:33 - of the high-resolution scene as we know it?
01:08:39 - Um it's an interesting question right it's it's
01:08:42 - something it it does make sense it's intuitive and in another sense it it seems
01:08:49 - a bit um homunculus um it seems like the screen that yeah as you said you you
01:08:55 - you can laugh about but you open the door but I told you I did,
01:09:00 - I did, but this is totally improved.
01:09:03 - I know. So I think, yes, we've won now looks to us as if you, um,
01:09:10 - um, you have kind of mental line drawings, um, uh, put together and we want,
01:09:18 - and maybe that's, that is, is, is a bit the language of V1.
01:09:20 - And I mean, it's an incredible story. I think if you look back, Peter, um.
01:09:30 - Patrick Kavanagh talks about this 10,000 years of neuroscience by describing
01:09:38 - the, if you look at the line drawings in caves,
01:09:45 - that is a form of communication that works.
01:09:49 - And that's surprising because line drawings work even though line drawings don't exist in the world.
01:09:56 - What we see or what's out there are textures and texture borders,
01:10:01 - but from a texture border to come to a line drawing is an abstraction that seems
01:10:08 - to work incredibly good for our visual system.
01:10:12 - And since we were totally surprised that if we asked the subjects to fill in
01:10:16 - the missing information, they came up with the same line drawing.
01:10:19 - And maybe there's a clue because um we find
01:10:23 - a communication that works it's so proved it's so
01:10:26 - error prone uh proven
01:10:29 - so so um you know if you
01:10:32 - you try to do an error a line drawing of an
01:10:35 - animal and um you're not happy with it
01:10:38 - you optimize it until you're happy and you think
01:10:41 - this is a good illustration of a of an
01:10:44 - um deer or
01:10:47 - so and then you present that to someone else and
01:10:51 - the value the quality and the the goodness of fit to their internal model is
01:10:57 - so immediate because we have the same kind of visual system and that opens up
01:11:01 - a communication if we would open up this kind of communication with monkeys
01:11:05 - you know they could start drawing out their internal models and you You know,
01:11:09 - that's interesting, right?
01:11:10 - This is a speculation we could have had in the 60s by saying,
01:11:13 - oh, the language of V1 are contrast boundary contours, right?
01:11:18 - So in some sense, what's interesting about this, as opposed to having sort of
01:11:22 - complex predictive hierarchy, maybe you need the majority of machinery of vision just sits in V1.
01:11:28 - And what these hierarchies are doing is basically, if you want,
01:11:31 - just in a very coarse way, modulating this process in V1.
01:11:35 - So the whole hierarchy is now collapsing into V1, essentially.
01:11:39 - And all the rest does is say, well, let's ignore these bits, right?
01:11:42 - Or, okay, let's attend more to that part. Without telling it precisely what
01:11:46 - it should be seeing, maybe that's sort of a mistake people make.
01:11:51 - What about the predictive model is therefore maybe not telling you exactly,
01:11:55 - you should see this texture in this position.
01:11:58 - It may be the prediction just tells you, look, there's something interesting
01:12:01 - there. Check it out, right? Yeah. Can you buy that? note?
01:12:06 - Um, yeah, I mean, there's, like...
01:12:10 - What are we also experts in is face processing, right?
01:12:13 - So if you can… Face with pH or… Oh, face with… No, the emotion.
01:12:22 - We can read out the emotion and the gender out of faces and we can see kinship
01:12:27 - relations and all these kind of incredible tasks in faces.
01:12:31 - Now, if you imagine the face of persons that you know very well,
01:12:37 - um your line drawings aren't
01:12:40 - very precise and and and it's it needs a
01:12:43 - lot of training to be good at that to to do the
01:12:46 - um happy phase of um your wife or um you know so from this but we see we have
01:12:58 - done experiment in which we see that the um encoding of emotions and gender
01:13:04 - discrimination makes differences in V1.
01:13:06 - So whether you have the task to recognize gender or to recognize the emotional expression.
01:13:12 - We mapped out retrotopic spaces around mouth and eyes, but you can see a feedback
01:13:18 - signal in the other parts that are still having a signature,
01:13:21 - whether it is happy or fearful and so on.
01:13:24 - So this, I think, doesn't speak to a feedback signal that is very precise,
01:13:30 - let's say, like a line drawing of a precise emotion of a face.
01:13:34 - But it's like a marker, a token that says, here's a face, should be happy,
01:13:42 - you know very well your daughter, and that is a prediction.
01:13:49 - Now, that is a good template. that if then the stimulus comes up and there's
01:13:54 - a change in the facial expression you have FFA and other the face network to
01:14:00 - work out the differences between your prediction and the actual input but the.
01:14:07 - Token, the marking of where something is expected, is negotiated with higher
01:14:14 - and earlier visual areas.
01:14:15 - So I think the idea that I triggered of a cave drawing, in a way,
01:14:21 - B1 is something like active blackboard, which works like line drawings,
01:14:27 - and tokens, maybe tokens adjusted to that, where you say, oh,
01:14:30 - bad animal, and here is a phase of something.
01:14:33 - By this token, this is a particular B1. No, no, not the content.
01:14:37 - More like, you know, we open up a channel that we speak to each other.
01:14:43 - So you need to be a channel of high spatial frequencies because you're now interested
01:14:47 - in the emotion around the mouth.
01:14:49 - And that is interesting in this part of V1.
01:14:52 - V1 doesn't know anything about those phases.
01:14:55 - But the token is in the communication and higher.
01:14:59 - I thought you put a token also inside V1. So that sounded confusing to me.
01:15:03 - Like this, I get it. Absolutely. Yeah.
01:15:05 - That makes sense, because then we have very parsimonious linking,
01:15:09 - but then it's not necessarily defined along this notion of hierarchies of foreign models.
01:15:15 - It's a rather different kind of framework we're then in.
01:15:20 - Well we're talking about the generative capacity of the brain
01:15:22 - now and how that can uh you know
01:15:25 - be a tool for thought you know whether you're imagination and
01:15:29 - dreaming all of these processes where there's no visual input uh you can reconstruct
01:15:35 - activity throughout the visual hierarchy and including in v1 if you want to
01:15:41 - think about the details of say a line drawing you can imagine and that might
01:15:45 - require or involve activity in V1,
01:15:48 - which then sets off cascades of activity elsewhere in the brain.
01:15:52 - So we don't have to invoke a homunculus to see why it's worth reconstructing
01:15:57 - the visual scene at the V1 level. Look, I appreciate that.
01:16:02 - Speaking up for Lars now. No, I'm not. This is very much my own idea about what
01:16:07 - we might, why V1 might operate in this way.
01:16:11 - Look, I agree with this. I don't have a problem with that.
01:16:15 - Okay. My remark with the criticism was a bit different.
01:16:19 - Like, we started with a rather explicitly defined notion of prediction hierarchies,
01:16:24 - which are all sort of convergent to something like a Kalman filter.
01:16:28 - It's very explicitly defined and it really dictates to you what should happen
01:16:32 - in these cascades of interactions.
01:16:34 - And what I think we've seen at the end of the discussion, that there are cascades
01:16:38 - of interaction, but they might
01:16:40 - not follow this very restricted view on hierarchies of predictive filters.
01:16:46 - We might have to open up that perspective more. Like how Lars now describes
01:16:50 - the idea of having a higher level area that's a very abstract token type representation
01:16:54 - of something and it seeks information in preceding areas is not necessarily
01:16:59 - following this idea of prediction hierarchies.
01:17:01 - Maybe it just looks for confirmation, doesn't give a damn about errors, right?
01:17:04 - And that's how you can hallucinate things because you don't care about errors.
01:17:08 - You just care about confirmation about whatever stuff you believe in higher areas.
01:17:12 - So that was one of my challenges. And maybe the data and discussion is leading
01:17:16 - us to the point that we should open up the perspective.
01:17:18 - These are not just strictly defined prediction hierarchies. There's not enough data to support that.
01:17:24 - And maybe this model of token representations are maybe an alternative that
01:17:29 - is richer and maybe also closer to data.
01:17:32 - That was basically what I was saying.
01:17:34 - So are you happy with that, Tony? Or do you think I'm sort of misconstruing the discourse now?
01:17:39 - I think you're pushing it. Which I happily do most of the time.
01:17:42 - You're very much pushing it in a certain direction which I don't think Lars
01:17:45 - was necessarily agreeing with. But he was not mishearing.
01:17:48 - Maybe he's just tired. I think that's probably the case and he has a plane to
01:17:52 - catch. You've drowned him down.
01:17:54 - Good, you see? And Gern wins today again.
01:17:59 - So, Mars, now, before you can escape, there's two last hurdles you have to take.
01:18:05 - The first one is, though, this is not easy territory.
01:18:10 - It's really involved experimental methods.
01:18:14 - It's linked to theory. It's very precise. It's hard work, right?
01:18:18 - This stuff doesn't come for free.
01:18:20 - So, I'm sure you have plenty of scars, being in this domain.
01:18:25 - So if we would like to follow your route to try to understand the brain, what is Lars' law?
01:18:31 - What is Lars' law that we should follow to understand the brain?
01:18:38 - Um okay so
01:18:42 - of course there is
01:18:46 - a kind of learning of
01:18:50 - the tools that we try
01:18:53 - to sharpen and to get better to explore
01:18:57 - so what we bring to the table and what
01:19:00 - every neuroscience labs brings to the a table or a
01:19:03 - different you know background is um expertise
01:19:08 - with the tools and um making
01:19:11 - them sharper being able to to explore more and more
01:19:13 - um cutting edge for example we do the ultra high resolution to layer specific
01:19:19 - fmi um and using this uh techniques of brain reading for non-stimulated areas
01:19:26 - this is something that we learned over time that is possible and it's there's
01:19:31 - a lot of methodological kind of question.
01:19:33 - Now, once you have this kind of toolset that you think is very good to explore,
01:19:39 - with vector topic mapping and individual subject analysis and all this, then it's important to,
01:19:49 - find.
01:19:54 - Relevant questions and explain them
01:20:00 - in such terminology that your corner field becomes bigger and wider and you
01:20:06 - find more and more overlap with other labs and…,
01:20:12 - This is a law. We have to be able to put it on.
01:20:16 - I don't know what you mean by law. I kind of understood what's my method.
01:20:20 - The law of effect, right?
01:20:25 - Or the law of how to do science. I'm talking more about how to do science.
01:20:32 - I think there's a lot that we start to learn by doing multidisciplinary.
01:20:41 - Multiscale, multi-method science, which
01:20:45 - is a very difficult thing to do because everyone is leaving a little bit of
01:20:50 - their comfort zone and a little bit of terminology that is very well and precisely
01:20:56 - defined to open up to other fields in which different terminology is applied
01:21:03 - and different problems.
01:21:05 - But by leaving a little bit of this comfort zone, the more we do that,
01:21:11 - the more we find overlap, the bigger questions we can tackle.
01:21:15 - It's something that certainly we have all experienced in the Human Brain Project
01:21:20 - where we come together and start discussing our different approaches at which
01:21:27 - you see incredible work,
01:21:29 - but it is when the incredible works of different labs come together and find
01:21:34 - common language that you trigger an enormous amount of synergy.
01:21:38 - And that's something that's very hard work and it's very difficult and not so
01:21:44 - comfortable and it's something very, very important in science.
01:21:47 - I mean, I'm sometimes surprised when you, for whatever reason,
01:21:51 - step outside of your field and experience some other science,
01:21:54 - be it during a review process, during a conference.
01:21:58 - Usually a conference you wouldn't usually go to, and you realize how fragmented science is.
01:22:05 - And so it is important to converge, to find bigger stories, to not fight only
01:22:13 - the corner to sharpen your methods,
01:22:15 - but also to step outside and get the bigger picture, how it converges and how it fits together.
01:22:20 - And there's a lot of synergy for many parts of science where I find it extremely exciting. Mm-hmm.
01:22:27 - There's a bit in the trend of Ortega Gazette, right? Who spoke of counteracting
01:22:32 - the barbarism of specialization.
01:22:34 - So something he was saying, don't get sucked into the specialization,
01:22:37 - keep an open mind, right? Look at the other surrounding domains.
01:22:41 - Yes, that's part of it. I'm also a fan of using your methods very precisely.
01:22:47 - I think brain imaging has been a bit scarred by rapid exploitation of stories.
01:22:56 - And sometimes it has a bad image.
01:22:58 - People have seen and believed images because they were on the brain and they
01:23:04 - looked so scientifically and the stories connected to those were not always
01:23:10 - linked to very hard and precise signs.
01:23:13 - And so I think it's very important to gain back some trust there.
01:23:17 - And, you know, I think we push the limits in this a lot in our lab. It's a lot of work.
01:23:23 - And I think others are doing this in their field as well.
01:23:27 - But then there's a lot of reward by opening up a little bit and seeing these
01:23:32 - discussions over dinner and lunchtime when everyone is suddenly starting to
01:23:37 - discuss other things outside of their scientific expertise.
01:23:42 - There's a lot of, as I said, synergy and new ideas and converging ideas, I think.
01:23:46 - Especially now that we're able to have a meringue that might reflect activity
01:23:49 - of estrocytes instead of nerve.
01:23:53 - But then the last question is uh you know
01:23:56 - the last time that tony bought me a beer or paid for
01:23:59 - a beer it's a long time ago and and
01:24:02 - this has to do with the fact that he lives actually in for
01:24:05 - a long time so you have me virtually neighbors right you're in glasgow not sure
01:24:11 - you're over that but for that reason tony will come visit you in four years
01:24:14 - in glasgow assuming We are still there and I'm sure you will be doing great
01:24:18 - work and it's going to just come in a notebook to check whether you have confirmed or falsified,
01:24:25 - a specific hypothesis that you're going to share with us today.
01:24:28 - So what's the hypothesis, the most critical hypothesis in your research program
01:24:34 - that you want to see tested in this four-year timeframe?
01:24:38 - From now on in four years, what I want to see tested… So you're going to have
01:24:42 - a shifting time window from now four years.
01:24:45 - I'm not going to give away my best ideas.
01:24:48 - You just have to make a prediction.
01:24:53 - What will happen in my field or in other fields I think we'll see a lot of,
01:24:58 - he's coming to your lab to check a specific so you're going to say and you will
01:25:04 - come as well and you will see that you'll get more than a beer we are very generous
01:25:11 - in Glasgow, we don't like Edinburgh,
01:25:15 - we don't have the same No,
01:25:18 - no So you're up for a prediction error there. Exactly.
01:25:25 - No, I don't have a good prediction in this.
01:25:28 - I mean, what I'm most intrigued, I think maybe four years is not going to be enough,
01:25:33 - but I want to see the double coding of internal models that are about current situations.
01:25:46 - They are predictive about the scene and
01:25:50 - the mental models about something else being simultaneously processed in our
01:25:56 - brains which we know and we know in which areas more or less but we don't know
01:26:01 - how these codes coexist without interfering with one another now having layer um um.
01:26:09 - Resolution fmi i believe we might
01:26:12 - be able to find different codes simultaneously in
01:26:15 - these different areas in the different layers and i
01:26:19 - have to disagree with the remark that you made earlier that the differences
01:26:22 - in the layers that they there aren't differences in the layers i think they
01:26:27 - are you know i said that it's more gradual there's more gradual yeah if it's
01:26:30 - not as discrete as these models yeah so we have some new data in which we we
01:26:36 - have um visual illusions that are.
01:26:39 - Explained away like motion-induced blindness which you
01:26:42 - see a different code is still there but it's
01:26:45 - not conscious so so there's more to come from this
01:26:48 - but the the double coding of
01:26:51 - mental imagery and visual predictions
01:26:55 - i think is something that i'm i'm interested it must come at some level we we
01:27:01 - and we might find some ways in which we can um see these double codes some Some
01:27:06 - for the internal models that are currently happening and then other ones that are counterfactual.
01:27:11 - And it's something that maybe human fMRI or human research is particularly well-tuned
01:27:17 - to because these kind of experiments are relatively difficult to do in animals.
01:27:22 - You would need to instruct them to do a visual process and simultaneously do a mental imagery task.
01:27:30 - It's not totally impossible, but that's something I think I want to have more
01:27:35 - knowledge about. How do you have these two codes simultaneously?
01:27:38 - That will be a long discussion over a lot of beers. Lars Möckli,
01:27:42 - thank you very much for this conversation.
01:27:44 - Thank you. Thanks for the invitation. It was a pleasure.
01:27:49 - The CSN podcast was produced by the Convergent Science Network of Biometrics
01:27:54 - and Biohybrid Systems, a project funded by the European 7th Research Framework Program.
01:28:02 - For more interviews, recorded lectures, or upcoming conferences in the field
01:28:08 - of biometrics and biohybrid systems, go to csnnetwork.eu.
01:28:14 - Music.
01:28:15 - And thank you for listening.

CSN Podcasts

Lars Muckli on predictive processing and visual cortex

Season 2019

Description

CSN Podcasts

Episodes

Yoram Vodovotz on inflammation and immune system

Vincent Hayward on haptics and touch

Viktor Jirsa on epilepsy and virtual brain

Stuart Wilson on self-organization and cortical maps

Marco Diana on addiction and dopamine

Timestamp

Be the first to leave a comment

Leave a commentCancel

Legal

Follow us on

Lars Muckli on predictive processing and visual cortex

Description

Episodes

Timestamp

Be the first to leave a comment

Leave a commentCancel

Tags

Legal

Follow us on

Login to enjoy full advantages

Please login or subscribe to continue.

Go Premium!

Enjoy the full advantage of the premium access.

Stop following

Cancel subscription