Listeners:
Top listeners:
play_arrow
Paul Verschure on consciousness and distributed adaptive control CSN Podcasts
play_arrow
Edvard Moser on grid cells and entorhinal cortex CSN Podcasts
play_arrow
Giacomo Rizzolatti on mirror neurons and action understanding CSN Podcasts
play_arrow
Robert Axelrod on game theory and prisoner's dilemma CSN Podcasts
play_arrow
Adrian Owen on disorders of consciousness and vegetative state CSN Podcasts
play_arrow
Jonathan Whitlock on markerless motion capture and posterior parietal cortex CSN Podcasts
play_arrow
Luis Puelles on neuroanatomy and prosomeric model CSN Podcasts
play_arrow
Zoltan Molnar on subplate neurons and cortical development CSN Podcasts
Episode 7 15.03.2013
play_arrow
PLAY EPISODE
What if planning is not about computing value functions but about performing probabilistic inference? Marc Toussaint shows how recasting optimal control as message passing opens new computational pathways for robotics and decision-making.
Subscribe for more from the Convergent Science Network podcast series.
Marc Toussaint presents a theoretical framework that reformulates planning and optimal control as probabilistic inference in graphical models. Rather than iterating backward through Bellman equations to compute value functions, his approach computes both forward and backward messages whose product yields a posterior distribution over actions. This shift in perspective is not merely notational: it leads to genuinely different approximation algorithms, particularly for complex problems like partially observable Markov decision processes and factored state spaces where traditional value function methods struggle.
The conversation traces the intellectual lineage from Kalman’s duality between control and filtering through Bert Kappen’s work on path integrals to Toussaint’s own generalization that operates over joint state-control processes without restrictive assumptions about dynamics or cost structure. A key theoretical achievement is demonstrating that many existing reinforcement learning algorithms emerge as special cases of this unified formulation, providing both theoretical elegance and inherited empirical validation.
Toussaint derives a model-free reinforcement learning algorithm from this framework where the policy is represented as a Boltzmann distribution. Analysis of its fixed-point properties reveals a surprising result: for non-optimal actions, the Boltzmann energy diverges to negative infinity, making them vanishingly improbable, while for optimal actions, it converges exactly to the optimal value function. The framework handles goal conflicts through the natural machinery of probabilistic inference, where inconsistent evidence simply reduces likelihood and the system finds probabilistic compromises.
The episode also explores Toussaint’s robotics applications, where model-based approaches using stochastic relational rules enable robots to generalize from minimal experience. Active exploration strategies that maximize information gain prove essential in the exponentially large state spaces created by relational representations of multi-object environments, allowing a robot that has observed balls rolling to intelligently seek out non-ball-shaped objects to test next.
Tagged as:
Framework graphical models Optimal Control planning as inference Probabilistic Inference Theoretical Value Functions
About the author call_made
Both the triumphs of humanity and its most evil deeds have resulted from collaboration. In a time where humanity is required to aspire to the former and minimize the latter, the question arises of how collaboration arises and why it fails. Surprisingly, this phenomenon, so central to who we are, is not well understood. Hence, a collaborative effort is required to understand collaboration in its full biological, psychological, sociological, cultural, and economic complexity and to translate this understanding into operational impact. This series of podcasts is one step toward achieving these complementary goals. The Collaboration Podcast presents interviews with people who are central orchestrators of collaboration in various domains including business, government, science, art, health, sustainability, and the military. The discussions were conducted by Prof. Dr. Paul F.M.J. Verschure and members of the Program Advisory Committee of the Ernst Strungmann Forum on Collaboration (https://www.esforum.de/forums/ESF32_Collaboration.html) during 2021 and had the goal to sketch a map of opportunities, challenges, and obstacles in human collaboration. The forum took place in May 2022, and now we would like to share this series of interviews with a broader audience. The full report of the Forum will be published in 2023 by MIT Press. The podcast was produced by the Convergent Science Network (https://www.convergentsciencenetwork.org/). Context: The stability of social systems depends critically on realizing sustainable methods of “collaboration,” yet how and by which means collaboration is achieved is not clearly understood; neither are the conditions or processes that lead to its breakdown or failure. Collaboration can be understood as cooperation between agents toward mutually constructed goals. Part of the reason for our lack of understanding is that the phenomenon of collaboration is, by nature, a highly multidisciplinary problem, and effective research into its complexities has been difficult to achieve across the broad range of scientific and technical disciplines involved. The need for a fundamental understanding of collaboration, however, has become increasingly important. Not only does humankind demand answers as it attempts to address critical challenges at multiple scales (e.g., climate change, migration, enhanced automation, social and economic inequality), but ever-increasing technological and economic means of interconnecting people and societies are disrupting long-established, familiar patterns of how we interact. Radical technological changes that are ongoing have the potential to reshape collaboration in ways that are currently hard to predict or influence (e.g., by altering configurations in interaction, information creation, and modes of communication). On one hand, such changes could disrupt hitherto stable forms of collaboration by affecting critical communication channels and traditional roles, as can be observed in the rapidly changing patterns in governance, commerce, and social interaction. Conversely, technology could lead to the emergence of novel, successful forms of collaboration that deviate from traditional “hierarchical” architectures. Evidence of this can be seen in areas as diverse as highly automated manufacturing plants, the open science movement, collaborative software repositories, user-centered services, and the sharing of economy-based modes of organization. Without a fundamental understanding of the mechanisms, processes, and boundary conditions of collaboration, it is not possible to evaluate or predict which of these possible scenarios are sustainable or even plausible. The Forum “How Collaboration Arises and Why it Fails” (May 8–13, 2022, Location: Frankfurt am Main, Germany) Chairs: Andreas Roepstorff and Paul Verschure Program Advisory Committee: Jenna Bednar, Julia R. Lupp, Bhavani R. Rao , Andreas Roepstorff, Ferdinand von Siemens, and Paul Verschure
15.03.2013
Exploring the convergence of neuroscience, robotics, and AI through conversations with leading researchers since 2010.
A project of the Convergent Science Network Foundation.
© CSN Podcasts. Developed by IMCreative & WEBC
✖
✖
Are you sure you want to cancel your subscription? You will lose your Premium access and stored playlists.
✖
Be the first to leave a comment