Application Research of Big Data Real-Time Processing Framework Based on Spark in Industrial Data Stream Analysis
Main Article Content
Abstract
To meet the real-time analysis requirements of massive heterogeneous data streams generated by the Industrial Internet of Things (IIoT) under Industry 4.0, especially in complex engineering environments such as electromagnetic testing, antenna measurement, microwave device monitoring, and communication equipment manufacturing, this paper proposes an industrial big data real-time processing framework based on Apache Spark. By integrating Spark Structured Streaming, Spark MLlib, and time-series data storage technology, a full-process architecture covering data collection, preprocessing, real-time computing, and visualization is constructed. Combined with three typical scenarios, including smart power plant equipment monitoring, new energy battery production, and intelligent machine tool processing, a multi-level performance improvement strategy is designed, involving dynamic partition pruning, Shuffle optimization, and edge preprocessing, including protocol parsing, preliminary outlier filtering, and data compression at the gateway level. The effectiveness of the framework is verified through multidimensional comparative experiments. The results show that, in scenarios with millions of collection points, the framework achieves a throughput of 89,000 records per second, controls end-to-end latency within 280 ms, improves equipment fault prediction accuracy to 94.2%, reduces storage costs by 70%, improves performance by more than 40% compared with traditional Spark solutions, and reduces comprehensive resource occupancy, including CPU, memory, and network I/O, by 25% compared with Flink solutions. This research provides a practical technical solution for industrial real-time data analysis and offers engineering reference for data-driven monitoring of electromagnetic test platforms, antenna measurement systems, and microwave communication equipment.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
W. Yuan, P. Deng, T. Taleb, J. Wan, and C. Bi, “An Unlicensed Taxi Identification Model Based on Big Data Analysis,” IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 6, pp. 1703-1713, 2016.
M. Huang, “Development Trend and Typical Applications of Industrial Big Data,” Telecommunications Science, vol. 32, no. 7, p. 4, 2016.
S. Ma, “Research on the Improvement of Military Combat Effectiveness by Big Data,” Technology Wind, no. 22, p. 1, 2019.
C. Li and Z. Fu, “Improvement of Naive Bayes Classifier,” Statistics & Decision, no. 21, p. 3, 2016.
F. Wei, J. Dong, and Q. Zhang, “Interpretation of Industrial Big Data White Paper (2017 Edition),” Information Technology & Standardization, no. 4, p. 5, 2017.
Q. Liu and S. Qin, “Prospect of Process Industrial Big Data Modeling,” Acta Automatica Sinica, vol. 42, no. 2, p. 11, 2016.
L. Liu, “Research and Application of Big Data Clustering Algorithm Based on Spark Platform,” Nanjing University of Posts and Telecommunications.
W. He and C. Shao, “Development and Challenges of Industrial Big Data Analytics Technology,” Information and Control, vol. 47, no. 4, p. 13, 2018.
Z. Wang, Practical Tutorial of Oracle Database 11g. Beijing: Tsinghua University Press, 2014.
S. Pan, “Research and Implementation of Cache and Fault-Tolerance Strategies for Distributed Computing Framework Spark,” Henan University.
L. Yang, Spark Big Data Real-Time Computing. Practical Development Based on Scala, 2022.
S. Dang, X. Liu, X. Wang, and C. Liu, “Design of Real-Time Data Collection and Analysis System Based on Spark Streaming,” Journal of Network New Media, vol. 6, no. 5, p. 6, 2017.
T. Li, “Research and Implementation of Test Data Processing System Based on Spark Streaming,” Xidian University, 2015.
X. Yu, “Design and Implementation of Real-Time Recommendation System Based on Spark,” Southeast University.
L. Dai, “Research and Implementation of Data Stream Sequence Pattern Mining Algorithm Based on Spark Streaming,” Beijing University of Posts and Telecommunications, 2018.
C. Ni, “Research on Application Status and Development Trend of Big Data Technology,” China Management Informatization, no. 024-016, 2021.
L. Tian, “Research on Big Data Analytics Platform Technology for Manufacturing,” Shandong University, 2017.
S. Gu, “Siemens MindSphere Promotes Digitalization Process,” Automation Panorama, vol. 34, no. 7, p. 3, 2017.
P. Waurzyniak, “TrendMiner Brings Advanced Analytics to Siemens’ Mindsphere,” Manufacturing Engineering, vol. 163, no. 1, p. 2, 2019.
M. Corinne, “ABB Ability Launches Smart Sensor for Motors,” Condition Monitor (TN.363), pp. 5-5, 2017.
X. Wang, W. Fu, Z. Ma, and Z. Wang, “Discussion on Intelligent Solution of Medium-Voltage Switchgear Based on ABB Ability,” China High-Tech, 2018.
W. Lyu, J. Chen, and J. Liu, “Intelligent Manufacturing and Global Value Chain Upgrading: A Case Study of Haier COSMOPlat,” Science Research Management, no. 4, p. 12, 2019.
Y. Zhao, “Aerospace Cloud Network INDICS Industrial Internet Cloud Platform: Empowering Manufacturing Enterprises in the Cloud Era,” Automation Panorama, no. 3, p. 4, 2018.
M. Cheng, “Industrial Internet Empowerment Platform: The Next Industrial Revolution,” Software and Integrated Circuit, no. Z1, 2019.
J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation, vol. 6, 2004.
X. Meng, J. Bradley, B. Yavuz, E. Sparks, and A. Talwalkar, “MLlib: Machine Learning in Apache Spark,” Journal of Machine Learning Research, vol. 17, no. 1, pp. 1235-1241, 2015.
Z. Zhou and F. Chen, “Overview of Big Data Real-Time Computing Platform Technology,” China New Telecommunications, vol. 19, no. 4, p. 1, 2017.
H. Qi, “Overview of the Development of Stream Processing Frameworks,” Informatization Research, vol. 45, no. 6, p. 8, 2019.
S. Chintapalli, D. Dagit, B. Evans, R. Farivar, and P. Poulosky, “Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming,” IEEE International Parallel & Distributed Processing Symposium Workshops, 2016.