DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization
ICLR 2024(2023)
摘要
Visual reinforcement learning (RL) has shown promise in continuous control
tasks. Despite its progress, current algorithms are still unsatisfactory in
virtually every aspect of the performance such as sample efficiency, asymptotic
performance, and their robustness to the choice of random seeds. In this paper,
we identify a major shortcoming in existing visual RL methods that is the
agents often exhibit sustained inactivity during early training, thereby
limiting their ability to explore effectively. Expanding upon this crucial
observation, we additionally unveil a significant correlation between the
agents' inclination towards motorically inactive exploration and the absence of
neuronal activity within their policy networks. To quantify this inactivity, we
adopt dormant ratio as a metric to measure inactivity in the RL agent's
network. Empirically, we also recognize that the dormant ratio can act as a
standalone indicator of an agent's activity level, regardless of the received
reward signals. Leveraging the aforementioned insights, we introduce DrM, a
method that uses three core mechanisms to guide agents'
exploration-exploitation trade-offs by actively minimizing the dormant ratio.
Experiments demonstrate that DrM achieves significant improvements in sample
efficiency and asymptotic performance with no broken seeds (76 seeds in total)
across three continuous control benchmark environments, including DeepMind
Control Suite, MetaWorld, and Adroit. Most importantly, DrM is the first
model-free algorithm that consistently solves tasks in both the Dog and
Manipulator domains from the DeepMind Control Suite as well as three dexterous
hand manipulation tasks without demonstrations in Adroit, all based on pixel
observations.
更多查看译文
关键词
Visual RL,Dormant Ratio
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要