Collaborative Optimization of Multi-Energy Multi-Microgrid System: A Hierarchical Trust-Region Multi-Agent Reinforcement Learning Approach

In the context of the expanding diversity of energy demands, an increasing number of heterogeneous Multi-energy Microgrids (MEMGs) are engaging in the collaborative framework of the Multi-energy Multi-microgrid System (MEMMG). However, following this trend, the existing centralized Integrated Energy Management System (IEMS) control strategy is unreliable for future energy systems, characterized by more complex optimization control and a flexible system structure. This paper introduces a hierarchical Multi-agent Deep Reinforcement Learning (HMADRL) approach for distributed IEMS in MEMMG. Firstly, by employing a hierarchical approach, this method simplifies control complexity by segmenting the overarching control challenge into tasks of collaborative planning and action control, which are distributed across varied multi-agent scenes. Considering both macro and microeconomic factors, alongside carbon emissions, the optimal operation of MEMMG is realized through a bottom-up edge multi-agent control approach, in contrast to traditional top-down centralized methods. Secondly, in the phase of the inter-MEMG collaborative strategy, the Centralized Training Decentralized Execution (CTDE) framework is adopted to overcome the problems of unstable training environments and large-scale agent training, and each heterogeneous microgrid can develop local strategies independently with the assurance that their internal data will not be overly exposed. Thirdly, within each MEMG, the Trust-Region (TR) model is introduced for multi-agent action control, adeptly addressing the effects of mutual exclusion in decision-making time series. Simultaneously, an initialized Hot Experience Pool (HEP) is implemented, efficiently reducing exploration in complex, high-dimensional spaces. Finally, the off-time agent model is integrated into the HMADRL environment and undergoes secondary training based on real interactions, thereby deriving the optimal energy management policy. The proposed method markedly reduces reliance on exact physical modeling systems. Comparative simulations validate the proposed control scheme’s efficacy.