A State-Decomposition DDPG Algorithm for UAV Autonomous Navigation in 3D Complex Environments

Over the past decade, unmanned aerial vehicles (UAVs) have been widely applied in many areas, such as goods delivery, disaster monitoring, search and rescue etc.. In most of these applications, autonomous navigation is one of the key techniques that enable UAV to perform various tasks. However, UAV autonomous navigation in complex environments presents significant challenges due to the difficulty in simultaneously observing, orientation, decision and action. In this work, an efficient state-decomposition deep deterministic policy gradient algorithm is proposed for UAV autonomous navigation (SDDPG-NAV) in 3D complex environments. In SDDPG-NAV, a novel state-decomposition method that uses two sub-networks for the perception-related and target-related states separately is developed to establish more appropriate actor networks. We also designed some objective-oriented reward functions to solve the sparse reward problem, including approaching the target, avoiding obstacles and step award functions. Moreover, some training strategies are introduced to maintain the balance between exploration and exploitation, and the network is well trained with numerous experiments. The proposed SDDPG-NAV algorithm is capable of adapting to surrounding environments with generalized training experiences and effectively improves UAV’s navigation performance in 3D complex environments. Comparing with the benchmark DDPG and TD3 algorithms, SDDPG-NAV exhibits better performance in terms of convergence rate, navigation performance, and generalization capability.