Optimal Control of Unknown Nonlinear Systems via a Multimodel-Based Multistep Reinforcement Learning Framework

Document Type

Article

Publication Date

9-19-2025

Department

Department of Mechanical Engineering-Engineering Mechanics

Abstract

Optimal control of unknown nonlinear systems is challenging due to the absence of the underlying dynamics model. Reinforcement learning (RL) has become an effective framework for such control by optimizing policies with measurement data of the system. However, RL often suffers from: 1) low sample efficiency due to the absence of domain knowledge (e.g., dynamics model); and 2) low learning efficiency due to slow propagation of the reward signal. Model-based RL methods and multistep RL methods were proposed to address these two challenges, respectively. It is natural to combine these two methods to take advantage of both benefits. However, this combination requires the learned single one-step dynamics model, commonly used in model-based settings, to perform multistep prediction in a recursive manner, leading to the error accumulation (also known as compounding error) problem. This work presents a multimodel dynamics learning framework to address the error accumulation challenge for general model-based multistep RL methods by circumventing the recursive prediction. We also present a specific multimodel-based multistep RL algorithm and validate it on benchmark nonlinear systems. It is shown that the multimodel framework improves multistep prediction of dynamics and that the presented multimodel-based multistep RL mostly outperforms model-free, single-model, and one-step counterparts, respectively. We also discuss the limitations of this work and potential future work.

Publication Title

IEEE Transactions on Artificial Intelligence

Link to Full Text

Share

COinS