Automatic Load Allocation in Power Dispatch Master Stations Using Multi-Agent Deep Deterministic Policy Gradient Algorithms
Main Article Content
Abstract
Automatic load allocation in power dispatch master stations has become increasingly challenging due to the uncertainty of renewable generation and the complexity of multi-regional power coordination. This study proposes an automatic load allocation framework based on a Multi-Agent Deep Deterministic Policy Gradient (MA-DDPG) algorithm to improve real-time dispatch efficiency and system adaptability. A distributed multi-agent architecture is established in which each dispatch region operates as an autonomous agent with coordinated communication and decision-making capabilities. By incorporating stochastic models for wind and photovoltaic generation together with a multi-objective optimization strategy that simultaneously minimizes dispatch cost, renewable energy curtailment, and load fluctuation, the proposed approach achieves coordinated global optimization under dynamic operating conditions. Furthermore, adaptive communication weighting and prioritized experience replay are introduced to enhance cooperation efficiency and accelerate policy convergence. Experimental results demonstrate that the improved MA-DDPG framework significantly enhances training efficiency, reduces operational costs, and strengthens load balancing robustness while maintaining stable system performance under renewable and demand uncertainties. The proposed methodology also provides a reliable optimization paradigm for communication-intensive smart grids, where electromagnetic information transmission and distributed sensing infrastructures support real-time coordination and intelligent energy management.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Advances in Neural Information Processing Systems, vol. 30, pp. 6379-6390, 2017.
J. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” Advances in Neural Information Processing Systems, vol. 29, pp. 2137-2145, 2016.
J. Zhang, H. Yang, X. Chen, and H. Li, “Multi-agent deep deterministic policy gradient for energy management in microgrids,” IEEE Transactions on Smart Grid, vol. 11, no. 6, pp. 5163-5173, 2020.
Y. Shi, Y. Li, N. Zhang, and C. Kang, “Distributed optimization for load dispatch in smart grids using multi-agent reinforcement learning,” IEEE Transactions on Industrial Informatics, vol. 15, no. 7, pp. 4210-4220, 2019.
X. Wang, Y. Wang, and J. Li, “MA-DDPG-based coordinated control of distributed energy resources in virtual power plants,” IEEE Transactions on Sustainable Energy, vol. 12, no. 3, pp. 1642-1653, 2021.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1402-1411, 2018, doi: 10.1609/aaai.v32i1.11694.
S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1587-1596, 2018.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1861-1870, 2018.
A. Rashidinejad, F. Li, and M. Shahidehpour, “Multi-agent reinforcement learning for distributed energy resource management under uncertainty,” IEEE Transactions on Power Systems, vol. 35, no. 4, pp. 2898-2908, 2020.
Y. Chen, Z. Yang, and J. Liu, “Multi-agent DDPG with attention mechanism for optimal load distribution in power dispatch centers,” IEEE Transactions on Industrial Electronics, vol. 69, no. 11, pp. 11842-11852, 2022.
H. Su, Y. Tang, and J. Zhang, “Distributed load dispatch in power grids using multi-agent deep reinforcement learning with communication constraints,” IEEE Transactions on Power Delivery, vol. 36, no. 4, pp. 2182-2192, 2021.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004, doi: 10.1109/TIP.2003.819861.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in Neural Information Processing Systems, vol. 27, pp. 2672-2680, 2014.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484-489, 2016, doi: 10.1038/nature16961.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.
L. Busoniu, R. Babuska, and B. De Schutter, “Multi-agent reinforcement learning: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 38, no. 2, pp. 156-172, 2008, doi: 10.1109/TSMCC.2007.913919.
C. Zhang, J. Wang, and Y. Li, “A multi-agent deep reinforcement learning approach for energy-efficient resource management in industrial IoT,” IEEE Internet of Things Journal, vol. 6, no. 5, pp. 8136-8146, 2019.
Y. Liu, F. Tang, and Z. Li, “Improved MA-DDPG with prioritized experience replay for automatic load distribution in power systems,” IEEE Transactions on Smart Grid, vol. 14, no. 2, pp. 1345-1355, 2023.
J. Zhao, D. Yang, and B. Chen, “Multi-agent reinforcement learning for coordinated voltage control in distribution networks with high PV penetration,” IEEE Transactions on Sustainable Energy, vol. 12, no. 2, pp. 987-998, 2021.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI), pp. 2094-2100, 2016, doi: 10.1609/aaai.v30i1.10295.
T. P. Lillicrap and D. Wierstra, “Stochastic neural networks for reinforcement learning,” in Proceedings of the 27th International Conference on Neural Information Processing Systems (NeurIPS), pp. 1298-1306, 2014.
C. Yang, Y. Zhang, and X. Wang, “Multi-agent deep reinforcement learning for distributed optimal power flow in distribution networks,” IEEE Transactions on Power Systems, vol. 35, no. 6, pp. 4706-4717, 2020, doi: 10.1109/TPWRS.2020.2987292.
W. Gao, J. Liu, and Y. Tang, “Multi-agent DDPG with adaptive communication for load balancing in power dispatch master stations,” IEEE Transactions on Industrial Informatics, vol. 18, no. 7, pp. 5012-5022, 2022.
X. Peng, K. Tang, and H. Li, “Distributed multi-agent reinforcement learning for dynamic load dispatch in smart grids,” IEEE Transactions on Power Delivery, vol. 36, no. 2, pp. 890-899, 2021.
Y. Wang, J. Zhang, and J. Li, “Multi-agent reinforcement learning-based energy management for microgrids with electric vehicles,” IEEE Transactions on Transportation Electrification, vol. 6, no. 3, pp. 1193-1203, 2020.
H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 11, pp. 2947-2960, 2019.
A. Rashidinejad and M. Shahidehpour, “Resilient multi-agent reinforcement learning for cyber-physical power systems,” IEEE Transactions on Smart Grid, vol. 10, no. 6, pp. 6904-6913, 2019.
Z. Chen, G. Li, and C. Wang, “Multi-agent deep deterministic policy gradient with dual-critic networks for power system load dispatch,” IEEE Transactions on Power Systems, vol. 38, no. 3, pp. 2567-2578, 2023.