Goal-Space Planning with Subgoal Models
arxiv(2022)
摘要
This paper investigates a new approach to model-based reinforcement learning
using background planning: mixing (approximate) dynamic programming updates and
model-free updates, similar to the Dyna architecture. Background planning with
learned models is often worse than model-free alternatives, such as Double DQN,
even though the former uses significantly more memory and computation. The
fundamental problem is that learned models can be inaccurate and often generate
invalid states, especially when iterated many steps. In this paper, we avoid
this limitation by constraining background planning to a set of (abstract)
subgoals and learning only local, subgoal-conditioned models. This goal-space
planning (GSP) approach is more computationally efficient, naturally
incorporates temporal abstraction for faster long-horizon planning and avoids
learning the transition dynamics entirely. We show that our GSP algorithm can
propagate value from an abstract space in a manner that helps a variety of base
learners learn significantly faster in different domains.
更多查看译文
关键词
Reinforcement Learning,Model-Based Reinforcement Learning,Planning,Temporal Abstraction
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要