Check out our latest work, "Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight," published in the IEEE Transactions on Robotics, where we reconcile
#OptimalControl and
#ReinforcementLearning, achieving the same super-human performance, but with superior generalizability, as our previous model-free deep RL! Code released!
PDF:
arxiv.org/pdf/2306.09852
Code:
github.com/uzh-rpg/acmpc_pub…
Full Video:
youtube.com/watch?v=_qekrF4E…
Model-free
#ReinforcementLearning (RL) is known for its strong task performance and flexibility in optimizing general reward formulations. On the other hand,
#ModelPredictiveControl (MPC) provides robustness, constraint handling, and powerful online replanning capabilities. In this work, we extend our previous AC-MPC paper (Romero, ICRA'24) by taking a deeper look at how both approaches can be unified. We introduce and extend Actor-Critic Model Predictive Control (AC-MPC), a framework that embeds a differentiable MPC inside an Actor-Critic RL architecture. This integration allows the MPC-based actor to perform short-term predictive optimization, while the critic facilitates long-horizon learning and exploration. We conduct a comprehensive study that highlights AC-MPC’s key advantages:
- Better out-of-distribution generalization, both against unknown disturbances and changes in the quadrotor dynamics
- Improved sample efficiency
- A novel empirical analysis uncovering a relationship between the critic’s value function and the MPC cost function, providing deeper insight into their interplay. We validate our method in simulation and the real world on a quadcopter flying at superhuman speeds of up to 21 m/s, matching state-of-the-art model-free RL performance, and retaining the predictive structure of MPC for more reliable out-of-distribution behavior.
Reference:
Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight
IEEE Transactions on Robotics (T-RO), 2025
PDF:
arxiv.org/pdf/2306.09852
Full Video:
youtube.com/watch?v=_qekrF4E…
Code:
github.com/uzh-rpg/acmpc_pub…
Kudos to
@roaguiangel,
@EliJalbout,
@realyunlong!
@UZH_en @UZH_Science @UZHspacehub @AUTOASSESS_EU @ERC_Research @UZH_ai