cover jonathan whitlock

Jonathan Whitlock on markerless motion capture and posterior parietal cortex

  • cover play_arrow

    PLAY EPISODE


cover jonathan whitlock
Season 2025
Season 2025
Description arrow_drop_down

Description

How do you track what an animal’s brain is doing when the animal itself is moving through space in complex ways? Neuroscientist Jonathan Whitlock from NTNU Trondheim describes the technical odyssey of building a markerless motion capture pipeline for rats, and explains why simplifying your behavioral paradigm can unlock deeper scientific insights. Subscribe for more from the Convergent Science Network podcast series. Jonathan Whitlock, who studies neural representations of posture and movement in the posterior parietal cortex, joins Paul Verschure and Tony Prescott at the Convergent Science Network’s Alicante Cognition, Brain and Technology Winter School. The conversation explores the practical challenges of tracking animal behavior with enough precision to decode neural signals, and how those challenges led Whitlock toward a radically simpler experimental approach: having rodents chase a visual target on a screen. The discussion opens with the technical hurdles of markerless motion capture. Whitlock’s lab spent years trying different marking methods, from tattoos to retroreflective paint to infrared pigments, before settling on marker-based tracking. Synchronizing neural recordings with postural data proved equally difficult, with months of data initially unusable due to insufficient temporal alignment. The payoff was substantial: discovering that even primary sensory areas encode body posture, something invisible without precise 3D tracking. The conversation then pivots to Whitlock’s new paradigm: a prey-chasing task where rodents pursue a moving dot on a screen, reinforced by medial forebrain stimulation. This approach collapses the behavioral problem to two variables, distance error and heading error, while tapping into innate predatory intelligence honed by evolution. Mice and rats learn the task rapidly with minimal training, demonstrating anticipatory behavior and strategic pursuit. The discussion draws connections to predation research using crickets, subcortical circuitry in the superior colliculus and amygdala, and the broader question of how to balance technical complexity against scientific clarity. Whitlock argues that the chasing paradigm opens access to forms of biological intelligence that have been optimized through natural selection, making it a goldmine for studying sensorimotor integration, prediction, and decision-making in freely behaving animals. Part of the Convergent Science Network podcast series from the BCBT Winter School.

Tagged as:

About the author call_made

CSN Podcasts

Both the triumphs of humanity and its most evil deeds have resulted from collaboration. In a time where humanity is required to aspire to the former and minimize the latter, the question arises of how collaboration arises and why it fails. Surprisingly, this phenomenon, so central to who we are, is not well understood. Hence, a collaborative effort is required to understand collaboration in its full biological, psychological, sociological, cultural, and economic complexity and to translate this understanding into operational impact. This series of podcasts is one step toward achieving these complementary goals. The Collaboration Podcast presents interviews with people who are central orchestrators of collaboration in various domains including business, government, science, art, health, sustainability, and the military. The discussions were conducted by Prof. Dr. Paul F.M.J. Verschure and members of the Program Advisory Committee of the Ernst Strungmann Forum on Collaboration (https://www.esforum.de/forums/ESF32_Collaboration.html) during 2021 and had the goal to sketch a map of opportunities, challenges, and obstacles in human collaboration. The forum took place in May 2022, and now we would like to share this series of interviews with a broader audience. The full report of the Forum will be published in 2023 by MIT Press. The podcast was produced by the Convergent Science Network (https://www.convergentsciencenetwork.org/). Context: The stability of social systems depends critically on realizing sustainable methods of “collaboration,” yet how and by which means collaboration is achieved is not clearly understood; neither are the conditions or processes that lead to its breakdown or failure. Collaboration can be understood as cooperation between agents toward mutually constructed goals. Part of the reason for our lack of understanding is that the phenomenon of collaboration is, by nature, a highly multidisciplinary problem, and effective research into its complexities has been difficult to achieve across the broad range of scientific and technical disciplines involved. The need for a fundamental understanding of collaboration, however, has become increasingly important. Not only does humankind demand answers as it attempts to address critical challenges at multiple scales (e.g., climate change, migration, enhanced automation, social and economic inequality), but ever-increasing technological and economic means of interconnecting people and societies are disrupting long-established, familiar patterns of how we interact. Radical technological changes that are ongoing have the potential to reshape collaboration in ways that are currently hard to predict or influence (e.g., by altering configurations in interaction, information creation, and modes of communication). On one hand, such changes could disrupt hitherto stable forms of collaboration by affecting critical communication channels and traditional roles, as can be observed in the rapidly changing patterns in governance, commerce, and social interaction. Conversely, technology could lead to the emergence of novel, successful forms of collaboration that deviate from traditional “hierarchical” architectures. Evidence of this can be seen in areas as diverse as highly automated manufacturing plants, the open science movement, collaborative software repositories, user-centered services, and the sharing of economy-based modes of organization. Without a fundamental understanding of the mechanisms, processes, and boundary conditions of collaboration, it is not possible to evaluate or predict which of these possible scenarios are sustainable or even plausible. The Forum “How Collaboration Arises and Why it Fails” (May 8–13, 2022, Location: Frankfurt am Main, Germany) Chairs: Andreas Roepstorff and Paul Verschure Program Advisory Committee: Jenna Bednar, Julia R. Lupp, Bhavani R. Rao , Andreas Roepstorff, Ferdinand von Siemens, and Paul Verschure

More posts

Timestamp

  • fast_forward00:00:04 - So can we aim to finish by 7.15
  • fast_forward00:00:08 - at the latest well that's not possible are
  • fast_forward00:00:12 - you have an appointment uh my wife here okay yeah i don't want to neglect my
  • fast_forward00:00:18 - wife involved me so i gotta put through well there you gotta come up with the
  • fast_forward00:00:24 - right answers fast well that's still 15 minutes 40 minutes. No, that's 45.
  • fast_forward00:00:30 - Okay, so we're at the Conversion Science Network podcast at our Alicante Cognition
  • fast_forward00:00:37 - Brain and Technology Winter School.
  • fast_forward00:00:40 - And so I'm here with Tony Prescott and Jonathan Whitlock from the University
  • fast_forward00:00:48 - of Technology in Trondheim.
  • fast_forward00:00:52 - And Jonathan, so you were sharing with us this morning,
  • fast_forward00:00:56 - actually this really incredibly complicated or challenging tour you went through
  • fast_forward00:01:01 - to analyze posture in rats, right, to also through this investment in new technology
  • fast_forward00:01:08 - for, in this case, behavioral tracking,
  • fast_forward00:01:11 - come up with new insights in how the brain is actually representing posture
  • fast_forward00:01:16 - and also using this kind of information in sensory processing and possibly behavior control.
  • fast_forward00:01:23 - So what what were the main technical challenges you were facing there when you
  • fast_forward00:01:29 - went through this whole trajectory of building up this pipeline that you're
  • fast_forward00:01:33 - that you're working with yeah so,
  • fast_forward00:01:36 - easily it was when we were we were trying to do marketless tracking before marketless
  • fast_forward00:01:40 - tracking was a thing so it you know it it pays to be kind of in a sweet spot
  • fast_forward00:01:45 - of the curve and you know where technology is at and what you can do so um,
  • fast_forward00:01:52 - Yeah, we sort of signed on for markerless motion capture with this,
  • fast_forward00:01:59 - a German company that specializes in tracking athletes.
  • fast_forward00:02:03 - And they thought it would be fun and different to try to take on tracking of animals.
  • fast_forward00:02:09 - So the software they had was, you know, at that point in time,
  • fast_forward00:02:15 - I guess it was the most advanced thing that you could get commercially.
  • fast_forward00:02:17 - But it was like i was saying it was like before uh before
  • fast_forward00:02:21 - the markerless sort of like you know wave the revolution had happened in the
  • fast_forward00:02:26 - in in neuroscience behavioral neuroscience so how to mark the animals up we
  • fast_forward00:02:31 - we we tried a lot of different things um from tattoos to using um,
  • fast_forward00:02:38 - retroreflective uh paint dabbing it on the animal's
  • fast_forward00:02:41 - skin um i even got
  • fast_forward00:02:44 - a retro or infrared reflective pigment that was
  • fast_forward00:02:47 - used as an additive of paint um so we
  • fast_forward00:02:50 - were trying all kinds of ways to like mark the animal's skin so
  • fast_forward00:02:53 - that um uh we had some sort of
  • fast_forward00:02:55 - way of tracking them in infrared or near infrared so that we can control and
  • fast_forward00:02:59 - manipulate the lighting without without disturbing the animals um so that was
  • fast_forward00:03:04 - certainly the the biggest challenge was just how to mark them up and we eventually
  • fast_forward00:03:08 - sort of gave up on the markerless approach and all with marker based and i would
  • fast_forward00:03:12 - I'd like to flip back to MarketLess now that that's possible.
  • fast_forward00:03:16 - But knowing what I know now, that will not come cheaply or freely.
  • fast_forward00:03:20 - So that would mean having a PhD student or postdoc,
  • fast_forward00:03:24 - devoted just just to doing that so it was really i
  • fast_forward00:03:27 - mean the really the biggest hurdle was just how to how to
  • fast_forward00:03:30 - mark the animals up um uh synchronizing
  • fast_forward00:03:34 - the tracking system with the recording system wasn't that big
  • fast_forward00:03:36 - of a deal um but then actually uh synchronizing
  • fast_forward00:03:40 - the the data trains after the fact proved to be um obviously
  • fast_forward00:03:44 - very crucial and we know you know synchronization is kind of
  • fast_forward00:03:47 - a big deal now and we we try to be very robust and
  • fast_forward00:03:50 - and having redundancy in the way that we synchronized the
  • fast_forward00:03:53 - sampling but um for several months
  • fast_forward00:03:56 - of time after we had recorded data we thought that it was still uh
  • fast_forward00:03:59 - we weren't able to see the signal because the the neural data and the postural
  • fast_forward00:04:05 - tracking were more sufficiently synced um yeah so we've learned a lot of technical
  • fast_forward00:04:11 - lessons along the way and you kind of gain kind of a meta knowledge of um uh.
  • fast_forward00:04:18 - What things uh feel worthwhile and what
  • fast_forward00:04:20 - things don't in terms of like how difficult it's going to be technically and
  • fast_forward00:04:24 - i had to say like after what i had my own lab for just over 10 years now um
  • fast_forward00:04:31 - i see a huge benefit in in reducing the complexity of of what you're tracking
  • fast_forward00:04:38 - and what you're recording and how you're going after it and so this is why i'm very very,
  • fast_forward00:04:43 - gravitated towards this chasing business.
  • fast_forward00:04:46 - You can take your whole question and project it onto a 2D surface and look at
  • fast_forward00:04:51 - the statistics of the movement of the target relative to the head.
  • fast_forward00:04:55 - So, you know, and you have basically a distance error and a heading error.
  • fast_forward00:05:01 - You can kind of collapse your whole problem down to these two variables while
  • fast_forward00:05:05 - the animal is really behaving.
  • fast_forward00:05:06 - And that's just incredibly appealing. Well, that's really interesting,
  • fast_forward00:05:10 - right? Because in some as you're saying, after mastering that problem of posture
  • fast_forward00:05:14 - tracking, because that's a difficulty, right?
  • fast_forward00:05:16 - It's three-dimensional posture that you have to extract.
  • fast_forward00:05:21 - After I've gone through all of that, and sometimes you're saying,
  • fast_forward00:05:23 - well, let's give up on that, because it's in some sense such a distraction from
  • fast_forward00:05:28 - the science you want to do.
  • fast_forward00:05:32 - Maybe let's simplify, right? That they can focus again on the scientific basics.
  • fast_forward00:05:37 - Right right so so what's that trade-off there between the
  • fast_forward00:05:40 - technical complexity and and the scientific
  • fast_forward00:05:44 - question you know i think the the art is
  • fast_forward00:05:46 - kind of finding the perfect balance of these things right so that
  • fast_forward00:05:49 - you can have the animal behaving naturally and
  • fast_forward00:05:53 - you're able to track its behavior as it's naturally expressed um you know you're
  • fast_forward00:05:58 - you're able to let the sort of like you know the intelligence of the pursuit
  • fast_forward00:06:02 - algorithms that are in the animal's head to be expressed like naturally through
  • fast_forward00:06:07 - the body um so as long as you're you know not interfering with that then you're.
  • fast_forward00:06:13 - Honestly the only reason right now
  • fast_forward00:06:16 - i think to to stay with the 3d tracking is because
  • fast_forward00:06:19 - that's what we have and because of that that's like the
  • fast_forward00:06:22 - lowest energy you know it's the lowest energy path to
  • fast_forward00:06:25 - goal if we want to change the
  • fast_forward00:06:28 - tracking setup now that's going to require someone to come in and you know
  • fast_forward00:06:31 - that that will require an active change um and
  • fast_forward00:06:34 - you know we could we could
  • fast_forward00:06:37 - do that but first and foremost i really want to get the uh
  • fast_forward00:06:40 - the software installed and i want to get the task up been going and then
  • fast_forward00:06:43 - he can you know set about central flying or yeah
  • fast_forward00:06:46 - determining the things that we really really want to go after the
  • fast_forward00:06:50 - data you're collecting justifies what you've done i think
  • fast_forward00:06:53 - because you're showing that even in
  • fast_forward00:06:55 - primary sensory areas we're getting signals that
  • fast_forward00:06:58 - are related to body posture absolutely uh you
  • fast_forward00:07:02 - wouldn't have been able to find out unless you have
  • fast_forward00:07:05 - this really detailed specific tracking of the position of the animal in space
  • fast_forward00:07:10 - yes so um you know I think and you didn't know when you set out that that would
  • fast_forward00:07:16 - happen but it demonstrated the value of that technology and I think there is
  • fast_forward00:07:23 - this trade off when I think.
  • fast_forward00:07:25 - Dell has to carry this head stage but you get very precise geometry out of it
  • fast_forward00:07:29 - so it's definitely a price worth paying.
  • fast_forward00:07:33 - So I agree because in some sense you can now make this step to 2D spot tracking
  • fast_forward00:07:39 - because you have understood the 3D posture.
  • fast_forward00:07:45 - Representation in the posterior parietal cortex.
  • fast_forward00:07:49 - So now you have a very solid foundation from which you can make that next step. Exactly.
  • fast_forward00:07:55 - And that's why I feel like this really is kind of like a new horizon. It feels...
  • fast_forward00:08:02 - It feels like more than, you know, just a dot on a screen that the animal's
  • fast_forward00:08:07 - chasing, because when you have closed-loop control and you,
  • fast_forward00:08:10 - the experimenter, have a means by which, you know, the animal thinks that chasing
  • fast_forward00:08:15 - this thing is fun, you can introduce so many different variables.
  • fast_forward00:08:19 - There's very, very many aspects of, of animal behavior and cognition that you gain access to.
  • fast_forward00:08:24 - Um, because you, you, you have a task where, um, the animal thinks that something is fun.
  • fast_forward00:08:29 - It's not motivated by, uh, you know, by a foot shock or thirst or fear.
  • fast_forward00:08:35 - It's, you know, it's engaged because it thinks that whenever it touches its
  • fast_forward00:08:38 - nose to the circle, it's, it's fun.
  • fast_forward00:08:40 - Um, so yeah, this, I mean, this really opens up a lot of doors.
  • fast_forward00:08:44 - So I, I, I feel like, um, yeah, I really had kind of a, I, I felt a little bit
  • fast_forward00:08:49 - struck after, uh, when the video I was showing this morning with a mouse, right?
  • fast_forward00:08:53 - I was, I was concerned that mice wouldn't be smart enough to learn to chase
  • fast_forward00:08:57 - the dog out of the screen. And I, I talked with a couple of lab members and
  • fast_forward00:09:01 - they, they were also pretty steppy.
  • fast_forward00:09:02 - They're like rats, maybe, but the rice, no, I don't think so.
  • fast_forward00:09:06 - And what I found is, you know, once the animals, um, you know,
  • fast_forward00:09:10 - you gain access to how smart they actually are.
  • fast_forward00:09:14 - What was also stunning to see in the case of the mouse,
  • fast_forward00:09:18 - at least the video showed us that there was also anticipatory behavior
  • fast_forward00:09:22 - just following people already anticipating where the dot would end that was
  • fast_forward00:09:27 - pretty astonishing it demonstrates it certainly so logically by the behavior
  • fast_forward00:09:31 - to go after because I know in the rat listing literature people spend months
  • fast_forward00:09:37 - training rats to discriminate texture.
  • fast_forward00:09:41 - So you think if it's going to take that long to learn, then maybe this isn't
  • fast_forward00:09:45 - something they're doing in the natural environment.
  • fast_forward00:09:47 - It's distinguishing rough and smooth with the buccus. Maybe that's something else.
  • fast_forward00:09:51 - But I think you're showing that with relatively little training,
  • fast_forward00:09:55 - you can capture a behavior that was,
  • fast_forward00:09:59 - And just speaking candidly, I think this is why it's a goldmine.
  • fast_forward00:10:04 - Yeah. I think it's a goldmine because you are tapping into a form of intelligence
  • fast_forward00:10:09 - that has been honed to the point where it's in their genes.
  • fast_forward00:10:12 - Right. This is a form of intelligence that's been optimized through eons of natural selection.
  • fast_forward00:10:17 - So it's not just like how smart is the mouse?
  • fast_forward00:10:20 - How smart is mouse A versus mouse B or C? How well did I train it?
  • fast_forward00:10:24 - What's its error rate in my task?
  • fast_forward00:10:27 - You're looking at like an innate form of biological intelligence so it's not
  • fast_forward00:10:32 - just the mouse that I'm querying it's really I see it as like I'm querying like
  • fast_forward00:10:37 - kind of like back in time like an evolution,
  • fast_forward00:10:40 - so that's why I'm very enthusiastic I'm very bullish on the chasing wouldn't
  • fast_forward00:10:46 - it mean so this is a chasing behavior so what do you think okay these would
  • fast_forward00:10:51 - be the pre-animals that the mice and the rats would go after it might be insects,
  • fast_forward00:10:55 - So if instead of a light blob, it would be an animated insect,
  • fast_forward00:11:02 - do you think it would affect the behavior in any way or it would be the same?
  • fast_forward00:11:08 - That's something that we can test empirically. But so this idea of using crickets
  • fast_forward00:11:16 - in the lab, as far as I know, is traceable back to a postdoc that was in Chris Neal's lab.
  • fast_forward00:11:22 - He's at University of Oregon. this postdoc was uh i
  • fast_forward00:11:26 - believe it was jennifer hoy um and
  • fast_forward00:11:29 - she has her own lab now in in utah where she's studying
  • fast_forward00:11:32 - uh predation and and predatory behavior and mice
  • fast_forward00:11:35 - that that learn that crickets are food so this
  • fast_forward00:11:38 - is something that she brought to the lab just by
  • fast_forward00:11:41 - playing around and at first it was like kind of a
  • fast_forward00:11:44 - disaster because the mice were scared of the crickets you know
  • fast_forward00:11:46 - they you know what's what's this thing and the the
  • fast_forward00:11:50 - trick was they found they had to leave the crickets in the home cage with the
  • fast_forward00:11:53 - mouse for a couple of days and once the mice learned that this
  • fast_forward00:11:55 - thing was food then they were off to
  • fast_forward00:11:58 - the races um so the thing is when you put a cricket in the arena with with the
  • fast_forward00:12:04 - mouse um this elicits this you know predatory instinctual uh you know hunt and
  • fast_forward00:12:10 - kill kind of instinct that has been um different cell types have been isolated
  • fast_forward00:12:14 - in uh based on neural projections between the amygdala and the brainstem.
  • fast_forward00:12:19 - So there are different phases of the hunt and the kill that have been isolated
  • fast_forward00:12:24 - where, you know, if you stimulate certain cells, the animal engages in hunting.
  • fast_forward00:12:29 - And if you stimulate another subclass of cells, they engage in,
  • fast_forward00:12:32 - it's this act of chewing, like biting specifically.
  • fast_forward00:12:37 - So, you know, in terms of the cell types that are involved and the subcortical
  • fast_forward00:12:43 - circuitry that's involved, um we we've gained access to that you know by studying
  • fast_forward00:12:48 - the the predation behavior which,
  • fast_forward00:12:51 - So that's around periectosal gray, which area we're talking about?
  • fast_forward00:12:54 - Yes. Colliculus. Right.
  • fast_forward00:12:56 - The colliculus is involved. That's the orienting bit, right?
  • fast_forward00:12:59 - Sorry? The colliculus would be the orienting part. Yes.
  • fast_forward00:13:02 - And actually, Jennifer, and I think it was from Jennifer Hoy's lab on her own,
  • fast_forward00:13:07 - like a part known and all this was still with Chris Neal.
  • fast_forward00:13:10 - They were looking at different subtypes of cells in the colliculus that were
  • fast_forward00:13:15 - responsible for different types of visual reorienting, like visual motor orientation behavior.
  • fast_forward00:13:21 - Um, so, you know, in terms of accessing the sort of subcortical circuitry that
  • fast_forward00:13:27 - is really important for, you know, releasing these instinctual predatory behaviors, um,
  • fast_forward00:13:33 - that's, I think that's, that is as far as you're going to get with a cricket
  • fast_forward00:13:37 - because, because, right, the cricket needs to be stochastic.
  • fast_forward00:13:41 - It's going to be trying to do whatever it can to evade this giant mouse that's
  • fast_forward00:13:47 - coming after it and it's going to eat it, right? So it needs to be unpredictable.
  • fast_forward00:13:52 - And so what the cricket gets you is this sort of model-free pursuit, right?
  • fast_forward00:13:59 - Where the cricket is moving randomly and the mouse is just reacting to that,
  • fast_forward00:14:03 - right? And doing whatever it can to reorient and respond as quickly as possible.
  • fast_forward00:14:08 - Um the thing with the the chasing paradigm that
  • fast_forward00:14:10 - i want to develop and and this time a year ago i was racking
  • fast_forward00:14:14 - my brain thinking through digging up old tricks that
  • fast_forward00:14:17 - we had tried before for trying to like elicit different behaviors um thinking
  • fast_forward00:14:21 - how in the world am i going to get a mouse or a rat to chase after something
  • fast_forward00:14:25 - over which i have control yeah right because the the key here is that if you
  • fast_forward00:14:33 - the human can control i was even thinking of like using little micro grubs,
  • fast_forward00:14:37 - Yeah, I think that the issue with the lights can be listed.
  • fast_forward00:14:40 - A lot of the time they'd be hunting a dog using just their whiskers.
  • fast_forward00:14:44 - So the pathological validity.
  • fast_forward00:14:47 - So Mikhail Brecht in Berlin did stuff with the Etruscan Shrew.
  • fast_forward00:14:51 - You probably know the animal. The mammal with the smallest brain of any terrestrial animal.
  • fast_forward00:14:56 - And it's a masterful hunter of crickets and insects.
  • fast_forward00:15:01 - And it has tiny eyes. In the dark.
  • fast_forward00:15:05 - Fantastic Vibrisci. and he was looking at how the brisci respond to the shape
  • fast_forward00:15:11 - of the different prey animals and then they would target their bite very specifically
  • fast_forward00:15:18 - to the part of the insect where they could catch it.
  • fast_forward00:15:24 - But I think that your setup has a lot more accessibility in terms of what you
  • fast_forward00:15:31 - can manipulate and how you can interrogate that system. So that is an example
  • fast_forward00:15:36 - of the trade-off, you know.
  • fast_forward00:15:38 - It would be nice if you had live insects, but it's so much less controllable.
  • fast_forward00:15:43 - It's so much farther. Well, but I'm not sure about that because you could have
  • fast_forward00:15:46 - a 2D controller with a little stick on which you attach the insect and it moves
  • fast_forward00:15:51 - through the environment, right?
  • fast_forward00:15:54 - And you have a living insect on the end of it, so it's flapping its wings and
  • fast_forward00:15:57 - whatever, making the sort of eclaturing invalid noises.
  • fast_forward00:16:01 - And you can still move it on any trajectory through that space,
  • fast_forward00:16:04 - right? It's what it's possible.
  • fast_forward00:16:06 - Yeah. So I mean, that was, that was, I mean, that was another possibility I
  • fast_forward00:16:09 - considered of having a bait that was stuck down on the arena. Um,
  • fast_forward00:16:17 - specifically because of this uh because it's it's three-dimensional it
  • fast_forward00:16:20 - makes sense to the it makes sense to the rat um right so you know there is a
  • fast_forward00:16:26 - a few you know different possibilities that yeah that i could try depends on
  • fast_forward00:16:30 - your question doesn't it so which bit of system do you want to interrogate well
  • fast_forward00:16:33 - so the the thing that i liked about the um the visual stimuli.
  • fast_forward00:16:39 - Things all changed when i got into the lab and actually
  • fast_forward00:16:43 - you know and this is fun like i you know uh was
  • fast_forward00:16:46 - doing these these pilot experiments with that first rat
  • fast_forward00:16:49 - myself um i didn't want
  • fast_forward00:16:52 - to have my own people stop their work so that they could you
  • fast_forward00:16:55 - know plan this like a little side project
  • fast_forward00:16:58 - for me i i was very driven and very curious um
  • fast_forward00:17:01 - the the changing point for
  • fast_forward00:17:04 - me was once i saw that i could reinforce
  • fast_forward00:17:07 - how how easy it was uh uh
  • fast_forward00:17:10 - to stepwise training
  • fast_forward00:17:13 - the animal first to go after a circular white piece of paper because that
  • fast_forward00:17:16 - has texture and smell um to attract
  • fast_forward00:17:19 - the animal to it with the with the medial forebrain stimulation um
  • fast_forward00:17:24 - you could very quickly substitute you
  • fast_forward00:17:27 - could take the white piece of paper out and then you have the white circle that
  • fast_forward00:17:30 - is on the computer you know on the computer screen surface um and you start
  • fast_forward00:17:36 - by moving it just you know five or ten centimeters at a time in the animal you
  • fast_forward00:17:40 - know they're naturally curious anyway once they learn that there is a visual
  • fast_forward00:17:43 - object there uh they will go after it um and it was um.
  • fast_forward00:17:50 - It was just the experience of it it felt like yeah i was playing with uh like
  • fast_forward00:17:56 - a string with a cat it felt like playing with a cat yeah where you have you
  • fast_forward00:18:00 - know something suspended on a ball and you're dangling the uh the treat out
  • fast_forward00:18:04 - in front of the animal and it's going after it so once i saw that Once the animal became aware,
  • fast_forward00:18:10 - once it became apprised of the fact that there is this disc here,
  • fast_forward00:18:14 - and when I tap my nose, it's fine.
  • fast_forward00:18:18 - It felt like having a fish on a line and I could pull the rat around the arena
  • fast_forward00:18:26 - in whatever pattern I wanted.
  • fast_forward00:18:28 - Well, I think once you're in the colliculus, it's an multimodal head-centered
  • fast_forward00:18:32 - reference frame. For sure.
  • fast_forward00:18:34 - And vision is going there, touch is going there, audition is going there.
  • fast_forward00:18:37 - So you can choose which of those modalities you want to use as the trigger.
  • fast_forward00:18:41 - And then once you've said the animals decided that's the target then it's going
  • fast_forward00:18:46 - to recruit the same downstream mechanisms.
  • fast_forward00:18:48 - Yes, I would think so. So yeah, but if you want to obviously look at the sensing,
  • fast_forward00:18:52 - then you have to look at... But there's another caveat there, right?
  • fast_forward00:18:56 - Because if you start to look at the sort of core behavior system,
  • fast_forward00:19:00 - and to talk in the terminology of Bjorn-Murker.
  • fast_forward00:19:06 - Then it's all downstream on the spherical clits. A
  • fast_forward00:19:09 - decorticated mouse or rat could do it yeah
  • fast_forward00:19:12 - i would think maybe not predictably well but that's the
  • fast_forward00:19:15 - question it's an answerable question yeah
  • fast_forward00:19:18 - that's right it's a very exactly question so so once
  • fast_forward00:19:22 - i saw that the rat would respond to this
  • fast_forward00:19:24 - this thing that i was dragging around the screen and then
  • fast_forward00:19:27 - i could you know even so i'm using it the animals
  • fast_forward00:19:30 - tethered right and this is one of the reasons why i backed
  • fast_forward00:19:33 - away from trying like with this um with the with the protruding
  • fast_forward00:19:36 - stick or the insect coming down that could potentially wrap around the uh the
  • fast_forward00:19:40 - table there um it could also introduce a tracking artifact so this was certainly
  • fast_forward00:19:45 - i mean this was like on the short list of the top three things i was going to
  • fast_forward00:19:48 - try um uh this technically simplest thing was the visual stimulus and once i saw that,
  • fast_forward00:19:56 - you know the animals engaged and was chasing i thought
  • fast_forward00:19:59 - okay this this does it and i kind
  • fast_forward00:20:02 - of the thing that i did you know i had to like convince my colleagues of this
  • fast_forward00:20:06 - at first right before i had videos i was like i promise it's real like the chasing
  • fast_forward00:20:10 - is real um the rat had gone in so i was having it go in like a teardrop shape
  • fast_forward00:20:15 - around an obstacle right it was like chasing the cursor as i was dragging it over the.
  • fast_forward00:20:20 - The thing's a giant TV. I had my laptop plugged into it.
  • fast_forward00:20:24 - And I, you know, I had the screen black and I just, I could control the mouse
  • fast_forward00:20:27 - with my finger, right? The location of the arrow.
  • fast_forward00:20:29 - And I was having the animal run in a teardrop shape around an obstacle I'd put in the box.
  • fast_forward00:20:35 - And then eventually the cable became tangled. And I thought,
  • fast_forward00:20:38 - oh, I have to stop that. And I thought, no, I don't.
  • fast_forward00:20:40 - I'll just drag the cursor the other way. And I had the animal actually run in
  • fast_forward00:20:44 - circles and it untangled itself.
  • fast_forward00:20:47 - And I thought, this is, I'm God. this
  • fast_forward00:20:50 - is this is so cool i can have the right do whatever i
  • fast_forward00:20:53 - want um now the the big
  • fast_forward00:20:56 - um there are all kinds of insights that
  • fast_forward00:20:59 - come when you're when you're when you have the privilege
  • fast_forward00:21:02 - of playing with an animal and it's playing back with you um one
  • fast_forward00:21:06 - was object permanence right that i could have the cursor
  • fast_forward00:21:09 - run under this occlusion and the animal's like huh what
  • fast_forward00:21:12 - am i going to do now and it it certainly came around to
  • fast_forward00:21:14 - the other side like looking looking for the tree a real
  • fast_forward00:21:18 - aha moment happened um when you
  • fast_forward00:21:21 - know i was running the thing that's familiar teardrop shape
  • fast_forward00:21:24 - and the thing is is is approaching the object and the
  • fast_forward00:21:27 - rat actually darted ahead like the mouse did yeah it darted ahead to where it
  • fast_forward00:21:31 - thought the arrow was going to go and i thought oh my god this thing has an
  • fast_forward00:21:34 - internal model like this little fucker is predicting where the arrow is going
  • fast_forward00:21:38 - to go and i can i can record that so so that was a big that was a big i was
  • fast_forward00:21:44 - like oh my god and it's it's predicting where it's going to go.
  • fast_forward00:21:47 - I have access to this when when you're using a visual stimulus as well on the
  • fast_forward00:21:51 - computer surface you can introduce blinks you can you can introduce occlusions
  • fast_forward00:21:56 - whenever you want right and this is i think quite powerful because if you have
  • fast_forward00:22:02 - your target that is moving in a predictable trajectory,
  • fast_forward00:22:06 - and you think what you know if i argue what i'm
  • fast_forward00:22:08 - recording is the animal's internally generated prediction of
  • fast_forward00:22:12 - where the target is i i think a very good experiment to test that is to have
  • fast_forward00:22:17 - the thing moving along a a known trajectory that the animal's familiar with
  • fast_forward00:22:21 - but the actual physical presence of the target the actual image blinks on and
  • fast_forward00:22:26 - off at whatever interval i i don't know let's say half a second yeah.
  • fast_forward00:22:31 - Uh if the animal actually is is uh
  • fast_forward00:22:34 - has you know if it's actually making an
  • fast_forward00:22:37 - internally generated sort of prediction of
  • fast_forward00:22:40 - this movement pattern i should be able to
  • fast_forward00:22:42 - make rate maps for uh cells
  • fast_forward00:22:46 - that encode target location regardless of
  • fast_forward00:22:49 - if the target was visible or not there right so i could take all
  • fast_forward00:22:52 - the times in the session when the target was on when the blink was on
  • fast_forward00:22:54 - all the time in the session when the glint was off and if
  • fast_forward00:22:57 - it's true and if it's a true blue internal model and not just
  • fast_forward00:23:00 - visual feedback that's driving it then i should see
  • fast_forward00:23:03 - a similar receptive field that you know the neuron is
  • fast_forward00:23:06 - responding to regardless of whether or not the thing is actually you know visible
  • fast_forward00:23:10 - or not i think that's quite powerful and there's a big literature there on frontal
  • fast_forward00:23:13 - eye fields yes in many different monkeys showing how in these tasks where you
  • fast_forward00:23:20 - have to remember target you need frontal eye fields.
  • fast_forward00:23:23 - If it's just the target that is there then you can jog with the collectivus.
  • fast_forward00:23:28 - But if the target is there then it disappears and then you have to make movement
  • fast_forward00:23:32 - and you need the frontal eye fields.
  • fast_forward00:23:34 - So the cortex is I think for these predictive tasks it's going to be critical.
  • fast_forward00:23:39 - And I also took inspiration from Benjamin Hayden.
  • fast_forward00:23:43 - His lab was at University of Michigan and I think he recently moved down to Houston.
  • fast_forward00:23:48 - But he had A Korean postdoc who had a couple of very cool papers where they
  • fast_forward00:23:53 - were looking at the frontal eye fields of monkeys as they were chasing these
  • fast_forward00:23:56 - like, they called them intelligent agents. And I was like, Oh, I can call mine in.
  • fast_forward00:24:02 - Yeah but you know it's basically a cursor that is
  • fast_forward00:24:05 - is fleeing you know from the animal the animal has to move
  • fast_forward00:24:08 - its eyes or no no i'm sorry it was moving a joystick i think
  • fast_forward00:24:10 - um and the closed loop closed loop closed loop and the things could have different
  • fast_forward00:24:17 - patterns of fleeing that had different levels of difficulty and they looked
  • fast_forward00:24:21 - at the uh the latency they were extracting the latency by which uh the neurons
  • fast_forward00:24:25 - were predicting you know how like how far ahead was the brain calculating
  • fast_forward00:24:29 - where the target was going to be, you know, based on, I forget if it was the
  • fast_forward00:24:32 - animal's hand movement or eye movement.
  • fast_forward00:24:36 - But yeah, I definitely took inspiration from that, for sure.
  • fast_forward00:24:41 - But now the chasing could be predation, but it could also be for conspecifics, for sure.
  • fast_forward00:24:48 - So are you considering both options, or do you want to focus on predation?
  • fast_forward00:24:53 - No, no, no, no, no, no. i i i think um i the fun thing is i i think i'm co-opting
  • fast_forward00:24:59 - the pursuit machinery whether it's for predation or play or whatever i think
  • fast_forward00:25:04 - i would i would sooner think i'm mimicking play behavior.
  • fast_forward00:25:08 - Um uh simply because of the joviality of the animals i mean they gallop which
  • fast_forward00:25:13 - is like the the gold standard of a happy rat in a task is when they gallop they
  • fast_forward00:25:18 - do this kind of like bunny hop when they're running.
  • fast_forward00:25:21 - And that was another like kind
  • fast_forward00:25:23 - of golden moment that I had when I saw the animal galloping afternoon.
  • fast_forward00:25:27 - Fun. Right. Such a win. Because when it means that the animal's engaged.
  • fast_forward00:25:33 - It means it's invested in the task.
  • fast_forward00:25:35 - And when the animals are invested, that means you can get lots of trials,
  • fast_forward00:25:38 - you can get lots of sampling.
  • fast_forward00:25:40 - Yeah, good behavior almost always equals good behavior.
  • fast_forward00:25:43 - So play, that would fall, if you follow the late Shark Punk Shep distinction,
  • fast_forward00:25:48 - he would see play as one of the stereotype behaviors, right?
  • fast_forward00:25:51 - At this level of the core behavior system. But if it's play,
  • fast_forward00:25:55 - you could argue it's not that much, if you want, goal-oriented.
  • fast_forward00:26:00 - It's a more restricted exploratory behavior.
  • fast_forward00:26:05 - So if you now analyze the more detail what you've seen so far,
  • fast_forward00:26:11 - is it predation, is it play?
  • fast_forward00:26:13 - So is it really if you have this anticipatory component for instance,
  • fast_forward00:26:18 - right? I've seen the evidence.
  • fast_forward00:26:21 - Yeah, exactly. That was very clear. So would that still qualify as play or are
  • fast_forward00:26:26 - we and moving into another domain of really also prediction-based predation?
  • fast_forward00:26:32 - Or how are we going to distinguish that?
  • fast_forward00:26:35 - Uh, at this stage, I don't know
  • fast_forward00:26:37 - yet because it's, you know, because it's a video game. It's artificial.
  • fast_forward00:26:41 - Um, so I, I don't know. I mean, maybe the, the answer to that question might
  • fast_forward00:26:46 - come with, um, I could, I could imagine, right.
  • fast_forward00:26:50 - Um, having an animal where you do a CFOS dependent expression of some sort of
  • fast_forward00:26:55 - looker, right. where you could have a cricket in the arena, right?
  • fast_forward00:26:59 - And look at one set of cells and then engage the animal in this pursuit task
  • fast_forward00:27:04 - afterwards and see if the same cells, right?
  • fast_forward00:27:10 - I'm thinking back in time a while. The way to do this back in the day,
  • fast_forward00:27:14 - back 20 years ago, would have been to look at fish, like fluorescence in situ
  • fast_forward00:27:20 - hybridization and look at where CFOS RNA was localized in the cell.
  • fast_forward00:27:24 - Well, I'm not sure what molecular methods are available now.
  • fast_forward00:27:27 - But I would, yeah, I would do something like that, right? So engage the animal
  • fast_forward00:27:30 - in true blueprintation and look at the, you know,
  • fast_forward00:27:34 - cortical, but more like the subcortical pathways that are involved,
  • fast_forward00:27:37 - then sequentially engage an animal in this video game.
  • fast_forward00:27:42 - Well, even if it is play rather than predation, then the sort of esological
  • fast_forward00:27:47 - analysis of behavior it would suggest that these motivational systems are recruiting the same low-level.
  • fast_forward00:27:58 - Appetitive behavioral systems. So you're playing, but you're also using the
  • fast_forward00:28:03 - same mechanisms for pursuit.
  • fast_forward00:28:06 - That would be the hypothesis.
  • fast_forward00:28:09 - Why would there be separate mechanisms? Because part of the role of play is
  • fast_forward00:28:15 - to prepare the animal for,
  • fast_forward00:28:18 - in life. Well, you would. Yeah, that's true. Okay, no, fair enough.
  • fast_forward00:28:23 - But you would expect a difference in the motivational systems.
  • fast_forward00:28:26 - Yeah, you would. If you're looking in, you know, sort of maybe Paranormal Doctoral
  • fast_forward00:28:32 - Grey or someone like that, or Basil Ghandi, you would expect different activities.
  • fast_forward00:28:36 - Or if you manipulate, let's say, the hunger level, you would expect to see at
  • fast_forward00:28:42 - least a difference in the execution of the behavior when the animal is satiated, who we play,
  • fast_forward00:28:48 - or hungry you would expect it let's say
  • fast_forward00:28:51 - to be more efficient in capturing target if it's hungry than
  • fast_forward00:28:54 - if it's playful you know i mean right so this conversation right here is i this
  • fast_forward00:29:00 - is how i i i it's just me enforcing my notion that this is a this is a great
  • fast_forward00:29:05 - paradigm to look to because there's so many questions you can ask right but
  • fast_forward00:29:09 - the other thing that that there's a corollary corollary of that,
  • fast_forward00:29:13 - would be if it's anticipatory.
  • fast_forward00:29:17 - So here, I've seen this blob make a certain trajectory at some times,
  • fast_forward00:29:22 - and now I know where it's going to end up, so I'm going to stand there.
  • fast_forward00:29:25 - This must be represented somewhere. So would you see that as a hippocampal episode?
  • fast_forward00:29:32 - Or what's your speculation on that? Yeah, so.
  • fast_forward00:29:38 - I got a large, I'm starting
  • fast_forward00:29:41 - with a very large tv screen um so
  • fast_forward00:29:45 - that when the trials are presented um i can randomize where in the arena this
  • fast_forward00:29:51 - happens and i want to be very careful about the trajectories that this thing
  • fast_forward00:29:56 - takes and that the animal is is genuinely and try i have to try to restrict
  • fast_forward00:30:00 - what's going on the systems,
  • fast_forward00:30:02 - you know that that the animal can use can can bring to bear on the task,
  • fast_forward00:30:07 - and make sure that it's visual motor and that it is what i think
  • fast_forward00:30:10 - it is because if you for example
  • fast_forward00:30:14 - do what i was showing this this morning with the mouse where you have
  • fast_forward00:30:16 - the thing that comes on the same place every time it follows the same pattern
  • fast_forward00:30:20 - every time it goes to the end point the animal can very rapidly acquire the
  • fast_forward00:30:23 - spatial location right it learns the spatial trajectory and it learns the end
  • fast_forward00:30:27 - point so i think i mean we'll see what happens but i would suspect that if i control for space,
  • fast_forward00:30:34 - well, and the thing is moving on the flop.
  • fast_forward00:30:37 - And it's just visual motor prediction that I'm engaging.
  • fast_forward00:30:40 - I don't know if that would involve the hippocampus or not. It well could.
  • fast_forward00:30:44 - Is it a social memory? No, an episode.
  • fast_forward00:30:49 - I think, right, so the way to get with the episode, the episodic component,
  • fast_forward00:30:55 - would be, I think, to have the thing come on in a predictable location, right?
  • fast_forward00:30:59 - So my point in talking about the big TV is I want to have kind of a stack of
  • fast_forward00:31:05 - questions that I ask in a controlled kind of sequence.
  • fast_forward00:31:08 - Sure. With the first one being visual motor and like hippocampal-dependent episodic
  • fast_forward00:31:12 - rapid spatial learning is something that I would, you know, that's like another
  • fast_forward00:31:16 - PhD student's PhD. Sure, of course.
  • fast_forward00:31:19 - No, because one reason, remember in the synthetic forager project,
  • fast_forward00:31:23 - we did this experiment with surreal penarch because
  • fast_forward00:31:25 - we had this prediction that dynamic objects would
  • fast_forward00:31:28 - also be represented in hippocampus so we built rat proof robots so they were
  • fast_forward00:31:35 - like panzer robots that they couldn't chew on and then they would move in an
  • fast_forward00:31:40 - arena in certain patterns and we really found good correlation with location
  • fast_forward00:31:45 - and movement of these dynamic,
  • fast_forward00:31:47 - objects in hippocampus which was rather surprising because you would imagine
  • fast_forward00:31:55 - that the trajectory or the dynamics would not necessarily be considered, but it was.
  • fast_forward00:32:00 - So that's why I was wondering whether this pattern you're making with this blob,
  • fast_forward00:32:05 - is that integrated in an episodic dynamic memory?
  • fast_forward00:32:11 - Because otherwise, if it's reactive, then it's exactly like you say.
  • fast_forward00:32:14 - It's more like place learning. Okay, it ends up here, so the rest I don't care about.
  • fast_forward00:32:19 - Maybe it's the whole trajectory that's captured. Yes.
  • fast_forward00:32:22 - Well, honestly, I know this is kind of a top-out answer, but I'd like to look at all of it.
  • fast_forward00:32:29 - I really would. That's a top-out. Yeah, sorry.
  • fast_forward00:32:33 - I really like the object concept idea, the object components,
  • fast_forward00:32:40 - because there's really interesting parallels to infant development because the
  • fast_forward00:32:44 - newborn infant doesn't make predictive-eye movements.
  • fast_forward00:32:48 - They just follow things, and they don't have object permanence either so they're
  • fast_forward00:32:53 - surprised when something,
  • fast_forward00:32:54 - So you have, I think, the predictive eye movements emerging about three months
  • fast_forward00:33:00 - or so, which corresponds with some of the cortical systems kicking in.
  • fast_forward00:33:04 - So you potentially have a model for the development of predictive eye pursuit in transdermans there.
  • fast_forward00:33:12 - The other thing why it's a good hypothesis, and this is a rationale for it,
  • fast_forward00:33:16 - but it's also the minimal hypothesis.
  • fast_forward00:33:18 - So you can identify the boundaries.
  • fast_forward00:33:21 - So that you can just move the end point for instance and see okay when does the prediction fail,
  • fast_forward00:33:27 - well if you start with a more complex interpretation of like old trajectories
  • fast_forward00:33:30 - being represented and so on testability becomes more of a challenge but then
  • fast_forward00:33:34 - Tony you're sketching a developmental trajectory for this yeah absolutely because I think.
  • fast_forward00:33:40 - It's very nice because you can experimentally manipulate the role of cortex in this and the owl,
  • fast_forward00:33:47 - and you can in some way you can recapitulate what's happening in human development
  • fast_forward00:33:53 - where you're born with most of your, you know, we heard yesterday from Zoltac,
  • fast_forward00:33:58 - basically you've got a very minimal cortical network, which is the subplate,
  • fast_forward00:34:03 - and the rest of the neurons are still migrating into place and wiring themselves
  • fast_forward00:34:08 - up in those first three months.
  • fast_forward00:34:09 - And in that period, you're, I think,
  • fast_forward00:34:12 - controlled philosophy by your red brain and then it's rapidly wiring itself
  • fast_forward00:34:18 - up when you can see that behaviorally in the ability of infants and to predict your eye difference.
  • fast_forward00:34:25 - Oh, I can see where that's going and look over there. And also,
  • fast_forward00:34:28 - oh, the thing went behind the screen and I know it's going to come out over here.
  • fast_forward00:34:31 - So these are really important developmental milestones, which you can then look at in your own mobile.
  • fast_forward00:34:36 - I'm already thinking about, you know, would this be possible?
  • fast_forward00:34:39 - The Neuropixels, too, are very, very small and very, very light.
  • fast_forward00:34:43 - You can use them with no problem on mice.
  • fast_forward00:34:46 - I'm thinking it probably would be able to work on a young rat.
  • fast_forward00:34:50 - So, uh, I'm not sure. I mean, you know, the, the, the, the, the trajectory of
  • fast_forward00:34:55 - development, you know, is certainly much more accelerated, uh,
  • fast_forward00:34:59 - in, in the, I'm just thinking about, you know,
  • fast_forward00:35:02 - they have eyeopening and they start whisking and exploring away from the mother,
  • fast_forward00:35:06 - like all around the same time between P13 and P15.
  • fast_forward00:35:09 - And this is right around like the on
  • fast_forward00:35:12 - the borderline of like where you can actually implant you know
  • fast_forward00:35:15 - a light probe a light probe and start start recording um i what i'm wondering
  • fast_forward00:35:22 - is if if that you know if the um uh predictive cortical machinery is is already
  • fast_forward00:35:28 - in place by then i'm not sure i mean it will possibly interact but certainly
  • fast_forward00:35:32 - because the eyes are open quite late.
  • fast_forward00:35:36 - And by the time that they're moving around exploring where to the nest around
  • fast_forward00:35:42 - that I think you can stay with the adult and use an activation or whatever to look at,
  • fast_forward00:35:49 - but I want to annoy Jonathan a bit more before we end up because we have been
  • fast_forward00:35:54 - fantasizing about this task quite a bit and it sounds really exciting but we
  • fast_forward00:35:59 - are positioning this whole experiment in an ecological niche that's not so red-like,
  • fast_forward00:36:04 - because it's a light blob, right?
  • fast_forward00:36:06 - And it's not playing out in the dark, the whiskers are not being used, and so on, right?
  • fast_forward00:36:13 - How can you generalize then from whatever you're going to find when the rat
  • fast_forward00:36:17 - is chasing lead blobs or the mouse is chasing mouse blobs to the more ecologically
  • fast_forward00:36:21 - valid context of, let's say,
  • fast_forward00:36:24 - predation in the dark using the whiskers, which is, of course,
  • fast_forward00:36:26 - more a proximal form of sensing and so on?
  • fast_forward00:36:29 - Well, I think, so Jason Kerr, his lab in Germany, they have,
  • fast_forward00:36:36 - they're like me, but even like,
  • fast_forward00:36:40 - to an even higher level where they're doing eye tracking and pretty moving mice.
  • fast_forward00:36:47 - They've done the cricket chasing task where the cricket is placed in the arena
  • fast_forward00:36:51 - and they were tracking the animal's eyes while it was engaging in the pursuit.
  • fast_forward00:36:56 - And they found that the animals actually do use vision.
  • fast_forward00:36:59 - They do use the sense of vision to hunt. If light is available, they will use light.
  • fast_forward00:37:04 - And what they do is they try to position it so that the prey item,
  • fast_forward00:37:09 - is in the sort of uh what i want to say the
  • fast_forward00:37:12 - temporal ventral sort of like field of view um that
  • fast_forward00:37:15 - is like down in front of the animal's nose so you know it seems
  • fast_forward00:37:18 - like in terms of vision that's the game is for the animal to do whatever it
  • fast_forward00:37:23 - needs to do to keep the field in this like you know ventral like towards the
  • fast_forward00:37:27 - mouth basically within that that field of view around them uh and then when
  • fast_forward00:37:31 - they get close enough to the animal uh to the prey item then they switch over to the whispers,
  • fast_forward00:37:37 - which is much more like a phobial kind of spatial feeling.
  • fast_forward00:37:40 - You have to be within a centimeter or so to actually detect it.
  • fast_forward00:37:44 - After that, you're relying on in the dark, you're going to use audition, but olfaction as well.
  • fast_forward00:37:50 - So, you know, olfaction is very cute. Yeah.
  • fast_forward00:37:55 - The angle is being quiet, but you can sniff it out.
  • fast_forward00:38:00 - But the other thing, I would like to switch back to the posture story before
  • fast_forward00:38:05 - we go to the finish line because.
  • fast_forward00:38:08 - I would have thought you would have been so super enthusiastic about the posture
  • fast_forward00:38:11 - results, but you sound more enthusiastic about your new task, right?
  • fast_forward00:38:15 - So you really left the posture analysis a bit behind. Well,
  • fast_forward00:38:18 - actually, I think these are also really very revealing and potentially groundbreaking
  • fast_forward00:38:24 - results because first you show that 3D posture is explicitly represented in
  • fast_forward00:38:29 - a somatotopic fashion, right?
  • fast_forward00:38:31 - Which was not known, but now we know, which is amazing.
  • fast_forward00:38:36 - But then on top of that you also show that neurons in primary sensory areas,
  • fast_forward00:38:43 - audition and vision also reflect posture, which is astonishing, right?
  • fast_forward00:38:49 - Because again, we see multimodal responses in these primary sensory areas,
  • fast_forward00:38:53 - but now representing body features, right?
  • fast_forward00:38:58 - So I think this is actually amazing because you're really working dogmas here, right?
  • fast_forward00:39:03 - We're moving away from the standard view of unimodal representations.
  • fast_forward00:39:07 - But then it raised that question, like, okay, what the hell are these cells
  • fast_forward00:39:11 - doing there? Is it a statistical anomaly?
  • fast_forward00:39:13 - Like, okay, we have convergence-divergence of projections, and some of them
  • fast_forward00:39:17 - happen to pick up somnismatosensory signals.
  • fast_forward00:39:21 - Or do you see it as really a part of a computational scaffold that helps the
  • fast_forward00:39:27 - processing in this area?
  • fast_forward00:39:29 - So how do you think about that? Yeah, yeah. I think, you know...
  • fast_forward00:39:36 - I guess I'll go back and blame it on Hugo and Liesl, right? So,
  • fast_forward00:39:39 - you know, their original recordings in Kittens were in the LGN.
  • fast_forward00:39:44 - Well, I don't want to say that. My knowledge of their work begins in 1959 when
  • fast_forward00:39:48 - they were in their 1959 work, and they were recording from the LGN and Kittens in Darkness.
  • fast_forward00:39:54 - And they saw, oh, there's so many.
  • fast_forward00:39:58 - The LGN, which we thought was visual, is so allied with activity in darkness
  • fast_forward00:40:02 - and appears to be modulated by when the animals are moving.
  • fast_forward00:40:06 - We need to get rid of this, right? If we're going to, if we're ever going to
  • fast_forward00:40:09 - understand vision, we have to get rid of all of this pesky behavior.
  • fast_forward00:40:12 - Um, and that kind of set a precedent, right? So, you know, if,
  • fast_forward00:40:16 - if there's a sensory system that you want to study, you do everything you can
  • fast_forward00:40:20 - to sort of lock the animal in place and isolate that one sense modality.
  • fast_forward00:40:23 - And that's very rational, but at the same time that has led to this,
  • fast_forward00:40:27 - um, how to say a very partial incomplete view of how brains compute. Yeah.
  • fast_forward00:40:36 - And the fact is, when vision happens, when audition happens, when you smell...
  • fast_forward00:40:44 - All the time in our life that's that is done either to
  • fast_forward00:40:48 - cause or affect movement yeah you're you
  • fast_forward00:40:51 - know uh if you if you smell something that smells good
  • fast_forward00:40:54 - then you immediately orient towards it right you have this sort of like sensory
  • fast_forward00:40:58 - gradient of of you know where is the source of the odor and then you you have
  • fast_forward00:41:02 - um if you have the postural and and sort of like um uh the body schema that
  • fast_forward00:41:09 - is being read up continually in cortex it provides a scaffolding,
  • fast_forward00:41:14 - that allows the individual to respond all the quicker.
  • fast_forward00:41:18 - And without even thinking about it, you can reorient towards something that
  • fast_forward00:41:22 - is good or you can reflexively reorient towards something that's bad.
  • fast_forward00:41:29 - Yeah, it's the same sort of way of thinking about the colliculus where you have
  • fast_forward00:41:34 - a map of sensory input and then you have a map on motor output.
  • fast_forward00:41:38 - What I think this is telling us is that this is also baked into cortex as well.
  • fast_forward00:41:42 - But would it account for dynamical features in the visual cortex that were so
  • fast_forward00:41:49 - far not understood? Have you
  • fast_forward00:41:51 - seen modulation of visual responses by posture, like a gain field? Yeah.
  • fast_forward00:41:56 - So that is something that I am concluding a journey where we sought out to answer just this.
  • fast_forward00:42:04 - And I had invested the time and effort and the the effort of a very,
  • fast_forward00:42:13 - very motivated postdoc who's been working for years to get this augmented reality dome going.
  • fast_forward00:42:21 - So the idea behind this previous wave of grant funding and effort was to record,
  • fast_forward00:42:30 - since we saw that there were posturally responsive cells in visual work.
  • fast_forward00:42:35 - The idea was to put the animal in an environment where we had complete control
  • fast_forward00:42:39 - over the animal's visual experience while it was freely moving.
  • fast_forward00:42:43 - And so I had this augmented reality dome built. This company called Warpalizer.
  • fast_forward00:42:49 - They're a Norwegian company and they build flight simulators.
  • fast_forward00:42:52 - And simulators for people to train to drive boats and barges.
  • fast_forward00:42:57 - And they thought that this would be a very cute, fun pet project to make a rat simulator.
  • fast_forward00:43:02 - So there were projectors that are closed loop. you
  • fast_forward00:43:05 - know synchronized with our tracking system um and
  • fast_forward00:43:10 - this system was built to accommodate neural pixels recordings as
  • fast_forward00:43:12 - well and the idea was to
  • fast_forward00:43:16 - um look at the the
  • fast_forward00:43:20 - we were going to have two very different emissions one is where the animal is
  • fast_forward00:43:25 - head fixed right in doing this sort of like traditional uh visual battery of
  • fast_forward00:43:31 - like moving stripes that are moving across the screen so the animal's head is
  • fast_forward00:43:34 - locked and the stripes are moving and the visual perception is happening passively.
  • fast_forward00:43:39 - The flip side is where the animal is in an arena and in this case the first
  • fast_forward00:43:44 - thing we were going to try was to have the stripes be static and we would do
  • fast_forward00:43:48 - gaze reconstruction and look at whatever the animal's eye swept over,
  • fast_forward00:43:53 - the stripes in the arena at different orientations to compare orientation selectivity
  • fast_forward00:43:59 - when the movement is induced by the animal versus when it is delivered past.
  • fast_forward00:44:04 - This is kind of like, you know, this long list of design that was brought into
  • fast_forward00:44:09 - effect. We were in the middle of doing this.
  • fast_forward00:44:11 - We were setting up to do this when, again, Chris Neal's lab at University of
  • fast_forward00:44:16 - Oregon, where he's studying active vision and using rodents as models,
  • fast_forward00:44:20 - used a much, much, much less sophisticated approach where they were doing eye
  • fast_forward00:44:26 - tracking with three moving animals.
  • fast_forward00:44:28 - And they used basically a machine learning
  • fast_forward00:44:31 - kind of patch that incorporated
  • fast_forward00:44:37 - the component of the movement to the animal and found that there is a multiplexing
  • fast_forward00:44:41 - that was at play when the animals were moving between the movement of the head
  • fast_forward00:44:47 - and the visual response to stimuli that they were moving in and out of the receptive fields.
  • fast_forward00:44:53 - So it was like a less how to
  • fast_forward00:44:56 - say glamorous a less um over the
  • fast_forward00:44:59 - top kind of way of doing what we were trying to do so that
  • fast_forward00:45:02 - paper came out in 2022 and within
  • fast_forward00:45:05 - six months of that my postdoc who knew enough linear
  • fast_forward00:45:08 - algebra and she was the coding wizard who was going to like sort of bring this
  • fast_forward00:45:12 - into life left for industry so it was a it was a double gut punch and um that's
  • fast_forward00:45:18 - tough it was very tough and uh i had some my postdoc and i had some very uh
  • fast_forward00:45:23 - i don't know words of choice.
  • fast_forward00:45:27 - Uh and we decided to reframe things and so we we basically dropped that work
  • fast_forward00:45:32 - of of you know looking at this this exact question you're asking um are there
  • fast_forward00:45:37 - are there game fields at play right it is revivable the work or not it's finished
  • fast_forward00:45:42 - we still have it down but the thing is i need the grant, I need the people.
  • fast_forward00:45:47 - The thing is, I think your data already speaks to the data you showed on the bowel cortex.
  • fast_forward00:45:54 - So with the methodology you have already, there's so many interesting questions
  • fast_forward00:45:58 - you can ask. So that's the thing.
  • fast_forward00:46:01 - I threw out a net.
  • fast_forward00:46:03 - Something survived and something didn't. This is...
  • fast_forward00:46:07 - With the scientific trajectory right there are all these missed opportunities
  • fast_forward00:46:11 - but at least you've got to hit a few of them sure absolutely
  • fast_forward00:46:14 - and the thing that i wanted to get at the reason
  • fast_forward00:46:18 - why i went after vision in the first place was because you
  • fast_forward00:46:20 - have the control of the angle of the stripe and
  • fast_forward00:46:24 - i thought okay that's much more precise than sound localization it's a stripe
  • fast_forward00:46:28 - you know where it is in the animal's visual field that has a certain angle it's
  • fast_forward00:46:31 - very crisp that's the only reason i wanted to go after vision i thought it was
  • fast_forward00:46:35 - the most accessible sense modality and as it It turns out tracking the whiskers, which,
  • fast_forward00:46:41 - you know, you would think is much, much harder, is working much better. Right.
  • fast_forward00:46:46 - Yes. With the head mounted cameras. And so we can start to get at this issue
  • fast_forward00:46:51 - of gene modulation of one system versus another, I think, in a very satisfactory
  • fast_forward00:46:57 - way because we have the postural tracking and we have the whisker posture at the same time. Okay.
  • fast_forward00:47:01 - Yeah, that's good. You're on your way for that, right? That's very cool.
  • fast_forward00:47:05 - Yeah. Okay. The thing that I like it because I can see a way to make it actually.
  • fast_forward00:47:10 - We can take these UMAPs, these low dimensional projections of the whisker deployment,
  • fast_forward00:47:16 - and we can look at the neural coding of that.
  • fast_forward00:47:19 - And we can very much filter out different postures, different running speeds,
  • fast_forward00:47:24 - different whatever, and to see if there is some sort of behavioral modulation
  • fast_forward00:47:28 - of the whisker representation.
  • fast_forward00:47:30 - And we can flip things around and put the whiskers first and see if there's
  • fast_forward00:47:34 - some sort of whisker-dependent amplification of certain postures.
  • fast_forward00:47:38 - Also, it looks to me, I mean, you said active sensing, but it's also...
  • fast_forward00:47:43 - It fits very nicely with predictive processing views of this is that actually
  • fast_forward00:47:48 - what the animal cares about is not the raw signal from the whisker it's what
  • fast_forward00:47:53 - it expects it to give from the whisker,
  • fast_forward00:47:55 - so and you know the whiskers is where it is because of the head and body movements
  • fast_forward00:48:01 - and and when you film for a rat's whisking you rapidly realize you know that
  • fast_forward00:48:06 - the actual whisker movement is such a small part of it,
  • fast_forward00:48:10 - it's about head placement and head position, head orienting.
  • fast_forward00:48:14 - Everything is coordinated. You can't interpret that signal except in the context
  • fast_forward00:48:19 - of knowing where all these other systems are.
  • fast_forward00:48:21 - And so, but the radical thing is, and this is where people in the bowel community
  • fast_forward00:48:27 - fall out, is, you know, what's happening in S1?
  • fast_forward00:48:31 - Is S1 really taking all these signals or is it happening further upstream?
  • fast_forward00:48:36 - And I think what your results speak to is actually, even in that primary area,
  • fast_forward00:48:42 - you have this convergence of all the relevant signals.
  • fast_forward00:48:46 - For interpreting the whisker input
  • fast_forward00:48:49 - yeah it's not just you know the deflection of the whisker
  • fast_forward00:48:52 - shot no no no so i think that there
  • fast_forward00:48:55 - is uh there's a level of behavioral integration that
  • fast_forward00:48:58 - you know we're we're seeing um it's very rare
  • fast_forward00:49:01 - um that you would see i don't think
  • fast_forward00:49:03 - we've ever seen a whisker uh whisker tuning
  • fast_forward00:49:07 - that that doesn't come with some sort of aspect of
  • fast_forward00:49:11 - of of head placement or running speed that comes with
  • fast_forward00:49:13 - it um that's to
  • fast_forward00:49:16 - say you we have tests like i was
  • fast_forward00:49:19 - uh trying to explain uh about how you can split your recordings let's say like
  • fast_forward00:49:25 - you have a tuning curve for a whisker posture when the whiskers are forward
  • fast_forward00:49:29 - and then you can you can split the data based on when the animal was running
  • fast_forward00:49:33 - versus when it wasn't uh to see you know if the tuning is genuinely conjunctive or not.
  • fast_forward00:49:39 - It's very, very rarely the case then that you see a tuning curve for a whisker
  • fast_forward00:49:43 - posture where some other aspect of behavior doesn't come with it.
  • fast_forward00:49:46 - Yeah, but the orthodoxy at least 15 years ago, I'm not sure where it is now,
  • fast_forward00:49:51 - is that if you look in the middle of a barrel at S1, you have cells which respond to whisker deflection.
  • fast_forward00:49:59 - And they're orientation tuned and so on, but they're not encoding the position of the head.
  • fast_forward00:50:08 - That would be happening upstream. I mentioned Michael Brecht earlier.
  • fast_forward00:50:14 - He also had some interesting results.
  • fast_forward00:50:17 - He showed that you could get responses in the bowel cortex when a rat was touching
  • fast_forward00:50:24 - another rat and it was different whether it was male or female,
  • fast_forward00:50:27 - whether it was alive or dead.
  • fast_forward00:50:29 - So this is, again, another radical idea, but these sort of social markers are
  • fast_forward00:50:35 - already there. That's what.
  • fast_forward00:50:37 - But is this because these high-level systems are then projecting back in a great
  • fast_forward00:50:43 - processing way their expectations about the sensory signal, or is it something about...
  • fast_forward00:50:49 - That goes back to Donald's earlier point, right, about the Hubel and Wiesel,
  • fast_forward00:50:53 - if you want, paradox, that by looking at the system, it's highly controlled.
  • fast_forward00:50:57 - Ways, you push it into a state space where it usually isn't,
  • fast_forward00:51:04 - right? Because it's not really behaving.
  • fast_forward00:51:06 - Everything is controlled. Everything is fixed. And now we want to,
  • fast_forward00:51:09 - and of course, then it looks like a feedforward system that just responds to
  • fast_forward00:51:11 - what the whiskers are doing.
  • fast_forward00:51:13 - Right. Well, under free moving conditions, it's a highly integrated system with
  • fast_forward00:51:17 - lots of top down predictions floating around and so on.
  • fast_forward00:51:20 - And so in some sense, now it's a surprise.
  • fast_forward00:51:23 - It actually shouldn't be a surprise. It's just that we put ourselves in a very
  • fast_forward00:51:26 - small corner of that space where brains actually usually are not living, right?
  • fast_forward00:51:32 - Yeah, I think Dora Anjelaki put it really well when she was saying we were both
  • fast_forward00:51:37 - voicing our views and opinions on virtual reality setups and how they can be very powerful.
  • fast_forward00:51:43 - But in her words, you're putting a brain in a state of conflict,
  • fast_forward00:51:47 - right, where you're providing feedback signals that it should be perceiving
  • fast_forward00:51:52 - and processing if it were freely moving.
  • fast_forward00:51:55 - And yet the vestibular signals, your sense of...
  • fast_forward00:52:01 - You know, propelling the body through space is absent. And so you have systems
  • fast_forward00:52:05 - that are, you know, out of conjunction.
  • fast_forward00:52:07 - They're a bit out of joint. So in her words, like you're studying the brain
  • fast_forward00:52:11 - when it's in a state of conflict.
  • fast_forward00:52:13 - So you're not really looking at how things are when they're intact.
  • fast_forward00:52:16 - You're looking at how it's coping with the fact that it's receiving some but
  • fast_forward00:52:20 - not all inputs that it should. Right. Yeah.
  • fast_forward00:52:25 - But to get to the finish line, two questions.
  • fast_forward00:52:29 - Tony likes to travel. So in three years time we go to Trondheim to visit you, maybe earlier.
  • fast_forward00:52:37 - But we're going to come there with a purpose. We come with a purpose because
  • fast_forward00:52:41 - you're going to see whether you have verified the one hypothesis that you're
  • fast_forward00:52:47 - most passionate about today.
  • fast_forward00:52:48 - So what's the one thing you want to see validated three years from now?
  • fast_forward00:52:53 - But that's a do or die thing, right?
  • fast_forward00:52:57 - What's that one hypothesis?
  • fast_forward00:53:01 - I'm sort of distracted by chasing at the moment. I mean, so what side point is this you're chasing?
  • fast_forward00:53:10 - I would be very, very satisfied simply to see that there is a receptive field
  • fast_forward00:53:18 - for a target even when the animal doesn't see it.
  • fast_forward00:53:21 - Right. And that it is, in fact, this internally generated sort of model that
  • fast_forward00:53:27 - the animal or that the brain is generating. But at what level of the neurexis?
  • fast_forward00:53:33 - Just thinking in parietal cortex. Maybe this isn't so... No,
  • fast_forward00:53:36 - I'm being very realistic.
  • fast_forward00:53:38 - But you'll find it in parietal. That's a good prediction. Exactly.
  • fast_forward00:53:42 - I think parietal, I think potentially in the FOF, the frontal orienting,
  • fast_forward00:53:48 - as College of Recall said. That's another place I would like to look.
  • fast_forward00:53:53 - But it's not so risky as I bought. We'll let you get away with it.
  • fast_forward00:53:57 - We'll let you get away with it.
  • fast_forward00:53:59 - Well, what I would really, really love to say, and I'll just jump right in the
  • fast_forward00:54:07 - crazy land here. That's what we like.
  • fast_forward00:54:09 - What I want to uncover with this task is the narrow basis of spatial attention.
  • fast_forward00:54:13 - Okay. Right? So what I think is that there is—I don't know what attention is.
  • fast_forward00:54:22 - I'll just call it a potentiated locus in space where you want to act.
  • fast_forward00:54:26 - And I would define it that way, as where you're looking to move next,
  • fast_forward00:54:30 - right? And I'll call that the locus of attention.
  • fast_forward00:54:33 - What I think is, you know, this, like, kind of came to me, like,
  • fast_forward00:54:37 - when I'm, like, waking up, so I'm half asleep.
  • fast_forward00:54:40 - There is, like, a local warping of space that happens, right?
  • fast_forward00:54:45 - It's sort of like, if I'm completely fixated on this iPad in front of me,
  • fast_forward00:54:50 - right, the rest of the world kind of falls away and I get tunnel vision,
  • fast_forward00:54:53 - right? I would like to record that somehow.
  • fast_forward00:54:57 - I would like to demonstrate that somehow, that there is a warping of receptive
  • fast_forward00:55:03 - fields that become enlarged, right?
  • fast_forward00:55:05 - As the animal is attending towards something, as it's wanting to go after something,
  • fast_forward00:55:09 - like you would have on a paperweight that kind of magnifies what's beneath it
  • fast_forward00:55:13 - as you slide it over a surface.
  • fast_forward00:55:15 - I would really, really love to access that.
  • fast_forward00:55:19 - That's very cool. Okay. The last question is, So you run your own lab for quite a while now.
  • fast_forward00:55:25 - You do very challenging experiments. And also you're in one of the world-leading
  • fast_forward00:55:29 - institutes in neuroscience with Edward and my Brit Moser in charge.
  • fast_forward00:55:37 - So if younger people, students, younger researchers would like to follow in
  • fast_forward00:55:43 - your footsteps, what's Jonathan's law that they should adhere to?
  • fast_forward00:55:49 - You have to aim for the stars if you want to get to the moon.
  • fast_forward00:55:54 - So, you know, dream big, go big.
  • fast_forward00:55:59 - And when you only get 10% of the way there, if you've drunk big enough,
  • fast_forward00:56:03 - then you're still doing well. Very good. That is very much what happened to me.
  • fast_forward00:56:08 - Very good. Well, Jonathan Whitlock, thank you very much for this conversation.
  • fast_forward00:56:11 - It's been a pleasure. Thanks for the discussion. Thank you. Thanks for the invitation.
  • fast_forward00:56:16 - And then you're still on time, Donnie, right? You're on a divorce now. Yeah, no.
  • fast_forward00:56:20 - Hot water. Exactly. That was great, really.
  • fast_forward00:56:26 - It was a lot of fun because we actually dealt with all the stuff that you didn't talk about.
  • fast_forward00:56:31 - Yeah, exactly. We were able to sort of pick up where I had to leave off.
  • fast_forward00:56:35 - Actually, one thing, we have done a lot of 2D tracking.

Be the first to leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

Convergent PozitivLinia Logo

Exploring the convergence of neuroscience, robotics, and AI through conversations with leading researchers since 2010.

A project of the Convergent Science Network Foundation.

© CSN Podcasts. Developed by IMCreativeWEBC

0%

Login to enjoy full advantages

Please login or subscribe to continue.

Go Premium!

Enjoy the full advantage of the premium access.

Stop following

Unfollow Cancel

Cancel subscription

Are you sure you want to cancel your subscription? You will lose your Premium access and stored playlists.

Go back Confirm cancellation