Analysis of Spurious Local Solutions of Optimal Control Problems: One-Shot Optimization Versus Dynamic Programming

Dynamic programming (DP) has a rich theoretical foundation and a broad range of applications, especially in the classic area of optimal control and the recent area of reinforcement learning (RL). Many optimal control problems can be solved as a single optimization problem, named one-shot optimization, or via a sequence of optimization problems using DP. However, the computation of their global optima often faces the NP-hardness issue due to the non-linearity of the dynamics and non-convexity of the cost, and thus only local optimal solutions may be obtained at best. Furthermore, in many cases arising in machine learning and model-free approaches, DP is the only viable choice, and therefore it is essential to understand when DP combined with a local search solver works. In this work, we introduce the notions of spurious local minimizers for the one-shot optimization and spurious local minimum policies for DP, and show that there is a deep connection between them. In particular, we prove that under mild conditions the DP method using local search can successfully solve the optimal control problem to global optimality if and only if the one-shot optimization is free of spurious solutions. This result paves the way to understand the performance of local search methods in optimal control and RL.