Keynote E: 9:00am
Alexandre Proutiere / KTH, Royal Institute of Technology.
Based on work with Bastien Dubail and Stefan Stojanovic
Abstract: Low-dimensional structure underlies many modern reinforcement learning (RL) algorithms. In steady state, Markov dynamics are trivially low-dimensional: after mixing, the dynamics are essentially rank one. But RL is concerned with finite-horizon problems, raising a natural question: how much low-dimensional structure is already present at finite horizons, before mixing?
We study this question through the successor measure. While the successor measure itself need not be approximately low-rank, we show that low-rank structure naturally emerges after shifting it by only a small number of transitions. We characterize both this emerging structure and our ability to learn it efficiently through a new notion of spectral recoverability. To control this quantity, we introduce Type II Poincaré inequalities, a new class of functional inequalities for Markov chains that connect low-rank structure to local mixing. These inequalities show how mixing at different spatial and temporal scales determines when a compact representation of the dynamics emerges. This perspective naturally reveals a hierarchy of timescales in Markov dynamics and leads to new hierarchical RL algorithms that exploit it.
Contributed Talks E
Keynote F: 2:00pm
Giorgia Ramponi / University of Zurich.
Reinforcement learning is moving from isolated agents with well-specified rewards toward systems in which agents interact with humans and with other adaptive agents. This shift makes familiar challenges, like reward misspecification, non-stationarity, credit assignment, and coordination, even more relevant.
In this talk, I will discuss how agents can learn when rewards are unavailable or incomplete, using demonstrations, human feedback, and interaction. I will focus in particular on multi-agent settings, where learning must account for strategic behavior and equilibrium, and connect recent results in multi-agent imitation learning and feedback integration to emerging challenges in agentic AI and real-world applications.
Contributed Talks F
Keynote G: 4:00pm
Christian Duriez / DEFROST, Inria, Univ Lille, CNRS, Ecole Centrale de Lille, Compliance Robotics, France
Reinforcement learning and model-based control are complementary approaches for soft and continuum robots. While RL has recently been very successfully applied to robotics, soft robots still pose unique problems: unconventional robot designs, rich contact interactions, underactuation, complex state representations, heavy simulation costs, and challenging sim-to-real transfer. Reinforcement Learning can be used to complement traditional control and design strategies and cover some of their limitations. In this talk, we will present SofaGym, a collection of Gym-style environments for training RL controllers on soft-robots, to discuss the specific challenges of soft robotics and how RL may help address them. We will then focus on an interventional cardiology example to show how Monte Carlo Tree Search can be used to explore nonlocal strategies and complement optimization-based inverse-model control. We will also present work on object manipulation using deformable grippers that combine hybrid approaches involving model-based control and learning. We will conclude by noting that observing nature leads us to consider that intelligence is broader than control, and in particular to develop methods for enhancing the robot’s physical—mechanical—intelligence, which simplifies control.
Pierre Schegg / CAMIN, Inria, LIRMM, Univ Montpellier, Montpellier, France
Reinforcement learning and model-based control are complementary approaches for soft and continuum robots. While RL has recently been very successfully applied to robotics, soft robots still pose unique problems: unconventional robot designs, rich contact interactions, underactuation, complex state representations, heavy simulation costs, and challenging sim-to-real transfer. Reinforcement Learning can be used to complement traditional control and design strategies and cover some of their limitations. In this talk, we will present SofaGym, a collection of Gym-style environments for training RL controllers on soft-robots, to discuss the specific challenges of soft robotics and how RL may help address them. We will then focus on an interventional cardiology example to show how Monte Carlo Tree Search can be used to explore nonlocal strategies and complement optimization-based inverse-model control. We will also present work on object manipulation using deformable grippers that combine hybrid approaches involving model-based control and learning. We will conclude by noting that observing nature leads us to consider that intelligence is broader than control, and in particular to develop methods for enhancing the robot’s physical—mechanical—intelligence, which simplifies control.