Clustering experience replay for the effective exploitation in reinforcement learning

•The limitation of the exploitation efficiency in existing reinforcement learning methods is analyzed in detail.•Clustering is combined into the experience replay by a divide-and-conquer framework to improve the exploitation efficiency.•Our experience replay can sufficiently replay all kinds of transitions in the current training with low time consumption.•A new reinforcement learning method is proposed to implement our experience replay.