Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning
arxiv(2024)
摘要
Multi-Agent Reinforcement Learning (MARL) algorithms face the challenge of
efficient exploration due to the exponential increase in the size of the joint
state-action space. While demonstration-guided learning has proven beneficial
in single-agent settings, its direct applicability to MARL is hindered by the
practical difficulty of obtaining joint expert demonstrations. In this work, we
introduce a novel concept of personalized expert demonstrations, tailored for
each individual agent or, more broadly, each individual type of agent within a
heterogeneous team. These demonstrations solely pertain to single-agent
behaviors and how each agent can achieve personal goals without encompassing
any cooperative elements, thus naively imitating them will not achieve
cooperation due to potential conflicts. To this end, we propose an approach
that selectively utilizes personalized expert demonstrations as guidance and
allows agents to learn to cooperate, namely personalized expert-guided MARL
(PegMARL). This algorithm utilizes two discriminators: the first provides
incentives based on the alignment of policy behavior with demonstrations, and
the second regulates incentives based on whether the behavior leads to the
desired objective. We evaluate PegMARL using personalized demonstrations in
both discrete and continuous environments. The results demonstrate that PegMARL
learns near-optimal policies even when provided with suboptimal demonstrations,
and outperforms state-of-the-art MARL algorithms in solving coordinated tasks.
We also showcase PegMARL's capability to leverage joint demonstrations in the
StarCraft scenario and converge effectively even with demonstrations from
non-co-trained policies.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要