Physical Information Fusion of Multi-modal Data for Online Fault Monitoring of Integrated Manufacturing

doi:10.3901/JME.260825

Abstract

Abstract: High-performance electronic equipment integrated manufacturing (IM) technology is essential for intelligent manufacturing. The stability and reliability of IM’ s monitoring are dependent on the offline detection after manufacturing, which make slate detection of the products and manufacturing process, resulting in the waste of human-machine-time and the potential loss caused by the stop work and production. Intelligent fault monitoring is essential, while lacking physical information fusion and contextual modeling capability and facing multiple challenges like difficult fully exploiting spatial and nonlinear dynamic features of signals. To address these limitations, a multimodal physical information fusion network (MPIF-Net) is proposed for online fault monitoring in IM-SLM processes. Multi-sensor deployment is employed to collect multimodal acoustic emission data, which are transformed into fused images containing spatial and nonlinear dynamic features while retaining the original temporal modality. A dual-attention convolution module and Mamba block are designed to capture both local and global information from the fused images. In parallel, a BiLSTM architecture is incorporated to extract temporal dependencies from the raw acoustic emission signals. In particular, a physics-informed neural network (PINN) is constructed to generate high-fidelity thermal modality signals, thereby improving monitoring accuracy. Finally, a channel-concatenation fusion strategy integrates physical information with data-driven features to achieve multimodal fusion. Experimental results demonstrate that MPIF-Net significantly outperforms existing methods across multiple monitoring metrics, achieving an improvement of 1.48% in the F1 score compared with conventional multimodal algorithms. The findings indicate that the method not only enhances the accuracy and reliability of fault monitoring but also provides a generalizable solution for intelligent monitoring in integrated manufacturing.

Key words: fault monitoring, selective laser melting, integrated manufacturing, multi-modal physical information fusion

CLC Number:

TG156

LÜ Ruichan, XU Han, ZI Bin, ZHANG Peifu, LI Yuji, HUANG Jing. Physical Information Fusion of Multi-modal Data for Online Fault Monitoring of Integrated Manufacturing[J]. Journal of Mechanical Engineering, 2026, 62(15): 142-153.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://www.cjmenet.com.cn/EN/10.3901/JME.260825

http://www.cjmenet.com.cn/EN/Y2026/V62/I15/142

References

[1] 蒋周明矩，熊异，王柏村. 面向工业5.0的人机协作增材制造[J]. 机械工程学报，2024，60(3)：238-253. JIANG Zhoumingju，XIONG Yi，WANG Baicun，et al. Human-machine collaborative additive manufacturing for industry 5.0[J]. Journal of Mechanical Engineering，2024，60(3)：238-253.
[2] FUNCH C V，PROUST G. Laser-based additive manufacturing of refractory metals and their alloys：A review[J]. Additive Manufacturing，2024，94：104464.
[3] ZHOU C，YAN X，WANG H，et al. Advancements in hydrogen embrittlement of selective laser melting austenitic stainless steel：Mechanisms，microstructures，and future directions[J]. Journal of Materials Science & Technology，2025，230：219-235.
[4] YIN Z，HOU J. Recent advances on SVM based fault diagnosis and process monitoring in complicated industrial processes[J]. Neurocomputing，2016，174：643-650.
[5] KHANZADEH M，TIAN W，YADOLLAHI A，et al. Dual process monitoring of metal-based additive manufacturing using tensor decomposition of thermal image streams[J]. Additive Manufacturing，2018，23：443-456.
[6] CHALESHTORI A E，AGHAIE A. A novel bearing fault diagnosis approach using the Gaussian mixture model and the weighted principal component analysis[J]. Reliability Engineering & System Safety，2024，242：109720.
[7] 雷亚国，贾峰，孔德同，等. 大数据下机械智能故障诊断的机遇与挑战[J]. 机械工程学报，2018，54(5)：94-104. LEI Yanguo，JIA Feng，KONG Detong，et al. Opportunities and challenges of machinery intelligent fault diagnosis in big data era[J]. Journal of Mechanical Engineering，2018，54(5)：94-104.
[8] 严如强，许文纲，王志颖，等. 航空发动机燃油控制系统故障诊断技术研究进展与挑战[J]. 机械工程学报，2024，60(4)：3-31. YAN Ruqiang，XU Wengang，WANG Zhiying，et al. Research status and challenges on fault diagnosis methodology for fuel control system of aero-engine[J]. Journal of Mechanical Engineering，2024，60(4)：3-31.
[9] 狄子钧，袁东风，李东阳，等. 基于多尺度-高效通道注意力网络的刀具故障诊断方法[J]. 机械工程学报，2024，60(6)：82-90. DI Zijun，YUAN Dongfeng，LI Dongyang，et al. Tool fault diagnosis method based on multiscale-efficient channel attention network[J]. Journal of Mechanical Engineering，2024，60(6)：82-90.
[10] XIE T，HUANG X，CHOI S K. Intelligent mechanical fault diagnosis using multisensor fusion and convolution neural network[J]. IEEE Transactions on Industrial Informatics，2021，18(5)：3213-3223.
[11] PU Y，ZHAO Y，ZHAO G，et al. Experimental study on laser assisted machining of silicon nitride ceramics based on acoustic emission detection[J]. Optics & Laser Technology，2025，180：111497.
[12] DU J，ZENG J，CHEN C，et al. Acoustic emission monitoring for damage diagnosis in composite laminates based on deep learning with attention mechanism[J]. Mechanical Systems and Signal Processing，2025，222：111770.
[13] FU S，ZOU L，WANG Y，et al. DCSIAN：A novel deep cross-scale interactive attention network for fault diagnosis of aviation hydraulic pumps and generalizable applications[J]. Reliability Engineering & System Safety，2024，249：110246.
[14] ZHANG S，LIANG W，ZHAO W，et al. Electro-hydraulic SBW fault diagnosis method based on novel 1DCNN-LSTM with attention mechanisms and transfer learning[J]. Mechanical Systems and Signal Processing，2024，220：111644.
[15] 温楷儒，陈祝云，黄如意，等. 基于可解释时空图卷积网络的多传感数据融合诊断方法[J]. 机械工程学报，2024，60(12)：158-167. WEN Kairu，CHEN Zhuyun，HUANG Ruyi，et al. Multi-sensor data fusion diagnosis method based on interpretable spatial-temporal graph convolutional network[J]. Journal of Mechanical Engineering，2024，60(12)：158-167.
[16] 陈其怀，林添良，马荣华，等. 基于多模态的履带式挖掘机自动驾驶决策网络[J]. 机械工程学报，2025，61(4)：380-388. CHEN Qihuai，LIN Tianliang，MA Ronghua，et al. Automatic driving decision network of crawler excavator based on multi-modal[J]. Journal of Mechanical Engineering，2025，61(4)：380-388.
[17] PETRICH J，SNOW Z，CORBIN D，et al. Multi-modal sensor fusion with machine learning for data-driven process monitoring for additive manufacturing[J]. Additive Manufacturing，2021，48：102364.
[18] XU H，LV R，ZI B. A multi-modal information fusion network combining CNN and adaptive Mamba for fault monitoring of selective laser melting ceramic printing instrument[J/OL]. IEEE Transactions on Instrumentation and Measurement，2025，doi：10.1109/TIM.2025.3585221.
[19] KHAN A，SOHAIL A，ZAHOORA U，et al. A survey of the recent architectures of deep convolutional neural networks[J]. Artificial Intelligence Review，2020，53：5455-5516.
[20] GU A，GOEL K，RÉ C. Efficiently modeling long sequences with structured state spaces[J]. arXiv preprint arXiv：2111.00396，2021.
[21] HOCHREITER S，SCHMIDHUBER J. Long short-term memory[J]. Neural Computation，1997，9(8)：1735-1780.
[22] WOO S，PARK J，LEE J-Y，et al. Cbam：Convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision (ECCV)，2018：3-19.
[23] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al. An image is worth 16x16 words：Transformers for image recognition at scale[J]. arXiv preprint arXiv：2010.11929，2020.
[24] ZHANG W，PENG G，LI C，et al. A new deep learning model for fault diagnosis with good anti-noise and domain adaptation ability on raw vibration signals[J]. Sensors，2017，17(2)：425.
[25] WANG W，LI X，XU Z，et al. Learning two-stream CNN for multi-modal age-related macular degeneration categorization[J]. IEEE Journal of Biomedical and Health Informatics，2022，26(8)：4111-4122.
[26] WANG L，ZHAO W. An ensemble deep learning network based on 2D convolutional neural network and 1D LSTM with self-attention for bearing fault diagnosis[J]. Applied Soft Computing，2025，172：112889.
[27] XIE J R，SAFDAR M，CHEN L Q，et al. Audio-visual cross-modality knowledge transfer for machine learning- based in-situ monitoring in laser additive manufacturing[J]. Additive Manufacturing，2025，101：104692.