Skill emergence and transfer in multi-agent environments.

A central problem in training artificial agents to perform complex skills is specifying appropriate cost functions whose optimization will lead to the desired behavior. Specifying detailed cost functions is laborious and often inefficient. The training of agents in competitive and cooperative multi-agent environments provides an avenue to circumvent these limitations: By competition and cooperation agents provide to each other a natural curriculum that can lead to the emergence of complicated skills, even if the rewards of the multi-agent game are simple [1]. Here we explore the emergence of complex strategies and skills in a simple hide and seek game simulated in a 3-D physics environment. We show that training using deep reinforcement learning (RL) leads to the emergence of several rounds of strategies, counter-strategies, and skills composed of several sequential behaviors. Our results suggest that training multiple agents in a sufficiently complex environment using large scale RL can lead to open-ended development of behavior. We furthermore show that emergent skills can be extracted and re-used in distinct environments. This skill transfer is both a useful evaluation metric for multi-agent emergence and suggests that multi-agent pre-training might be a useful strategy to generate targeted skills.