Optimal scheduling strategy of electricity and thermal energy storage based on soft actor-critic reinforcement learning approach

The energy management of a community-scale microgrid involves scheduling hybrid energy storage to balance both surplus and deficit in the electric power market. Traditional community scale microgrid economic scheduling is a model-based approach that relies on accurate system parameter and uncertainty prediction. This paper presents a data-driven reinforcement learning approach for community-scale microgrids with hybrid energy storage. The method employed is the Soft Actor-Critic (SAC), an actor-critic, off-policy, stochastic method with built-in entropy maximization that balances exploration and exploitation. The developed SAC-based approach is applied to the operation of electrical and thermal energy storage units with time-of-use electricity prices and stochastic renewable energy generation. A case study of community-scale microgrids employing real electricity and heat demand is presented. The simulation results show that SAC algorithm has better performance than other reinforcement learning methods in different scenarios, and the average training time is reduced by at least 3.7 %. The proposed storage management scheme reduces the average daily operation and maintenance cost by over 10 % in summer, and over 20 % in winter, respectively. At the same time, MG scheduling based on SAC algorithm can realize millisecond level scheduling. The results show that the algorithm can avoid constraint violation, improve to the optimal solution continuously, and realize real-time scheduling. Simulation results verify the effectiveness of the proposed method.