Knowledge Distillation Method for Compressing Large Language Model of Power Risk Identification and Improving Deployment Efficiency
Main Article Content
Abstract
Large language models demonstrate high precision in power-system risk identification; however, their massive parameter counts and high resource consumption hinder real-time deployment on resource-constrained edge devices used in electromagnetic sensing systems, wearable monitoring terminals, and compact power-monitoring devices. Achieving low-latency processing in distributed sensing structures and edge-based electromagnetic monitoring devices requires a significant reduction in computational overhead to ensure immediate detection of electrical hazards, abnormal equipment states, and potential power-system risks. This paper proposes a lightweight compression method based on hierarchical supervised knowledge distillation. Experiments show that, after applying the proposed knowledge distillation method, the inference latency is reduced from 235 ms to a minimum of 26 ms, which is better than DistilBERT’s 35 ms. The number of student model parameters is reduced to 4.3% of the teacher model, namely 14.5M versus 340M, while the classification accuracy reaches 89.4%, close to the teacher model’s 92.7%. The F1 score for the equipment failure category reaches 90.3%, verifying the efficiency and practicality of the lightweight model in resource-constrained scenarios involving power-risk identification, electromagnetic sensing, and edge-based intelligent monitoring.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
A. H. Huang, H. Wang, and Y. Yang, “FinBERT: A large language model for extracting information from financial text,” Contemporary Accounting Research, vol. 40, no. 2, pp. 806-841, 2023, doi: 10.1111/1911-3846.12832.
M. C. Rillig, M. Ågerstrand, M. Bi, K. A. Gould, and U. Sauerland, “Risks and benefits of large language models for the environment,” Environmental Science & Technology, vol. 57, no. 9, pp. 3464-3466, 2023, doi: 10.1021/acs.est.3c01106.
S. Majumder, L. Dong, F. Doudi, Y. Cai, C. Tian, D. Kalathil, et al., “Exploring the capabilities and limitations of large language models in the electric energy sector,” Joule, vol. 8, no. 6, pp. 1544-1549, 2024, doi: 10.1016/j.joule.2024.05.009.
M. J. Davenport, “Enhancing Legal Document Analysis with Large Language Models: A Structured Approach to Accuracy, Context Preservation, and Risk Mitigation,” Open Journal of Modern Linguistics, vol. 15, no. 2, pp. 232-280, 2025, doi: 10.4236/ojml.2025.152016.
W. Cain, “Prompting change: Exploring prompt engineering in large language model AI and its potential to transform education,” TechTrends, vol. 68, no. 1, pp. 47-57, 2024, doi: 10.1007/s11528-023-00896-0.
L. Fan, L. Li, Z. Ma, S. Lee, H. Yu, L. Hemphill, et al., “A bibliometric review of large language models research from 2017 to 2023,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 5, pp. 1-25, 2024, doi: 10.1145/3664930.
Q. Zhang, K. Ding, T. Lv, X. Wang, Q. Yin, and Y. Zhang, “et al,” Scientific large language models: A survey on biological & chemical domains. ACM Computing Surveys, vol. 57, no. 6, pp. 1-38, 2025, doi: 10.1145/3715318.
B. C. Das, M. H. Amini, and Y. Wu, “Security and privacy challenges of large language models: A survey,” ACM Computing Surveys, vol. 57, no. 6, pp. 1-39, 2025, doi: 10.1145/3712001.
H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, et al., “Explainability for large language models: A survey,” ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 2, pp. 1-38, 2024, doi: 10.1145/3639372.
R. Navigli, S. Conia, and B. Ross, “Biases in large language models: Origins, inventory, and discussion,” ACM Journal of Data and Information Quality, vol. 15, no. 2, pp. 1-21, 2023, doi: 10.1145/3597307.
A. Kai, L. Zhu, and J. Gong, “Efficient Compression of Large Language Models with Distillation and Fine-Tuning,” Journal of Computer Science and Software Applications, vol. 3, no. 4, pp. 30-38, 2023, doi: 10.5281/zenodo.15165118.
J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789-1819, 2021, doi: 10.1007/s11263-021-01453-z.
J. Lin, X. Dai, Y. Xi, W. Liu, B. Chen, H. Zhang, et al., “How can recommender systems benefit from large language models: A survey,” ACM Transactions on Information Systems, vol. 43, no. 2, pp. 1-47, 2025, doi: 10.1145/3678004.
S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu, “Unifying large language models and knowledge graphs: A roadmap,” IEEE Transactions on Knowledge and Data Engineering, vol. 36, no. 7, pp. 3580-3599, 2024, doi: 10.1109/TKDE.2024.3352100.
Y. G. Xu, X. P. Qiu, L. G. Zhou, L. G. Zhou, and X. J. Huang, “Improving bert fine-tuning via self-ensemble and selfdistillation,” Journal of Computer Science and Technology, vol. 38, no. 4, pp. 853-866, 2023, doi: 10.1007/s11390-021-1119-0.