Research on Multi-modal Segmentation Method for Surface Defects of Cold-rolled Steel Strips Considering Vibration Effects

doi:10.3901/JME.260339

Abstract

Abstract: To meet the current demand for high-precision online defect detection of cold-rolled strips, a cold-rolled strip surface multi-modal detection platform is designed and built, enabling synchronous acquisition of two-dimensional RGB images and three-dimensional point clouds of the strip surface. To address distortions in three-dimensional point clouds caused by vibration and strip-profile interference under production conditions, a depth map transformation method considering vibration interference is proposed and is composed of three modules: coordinate projection transformation, surface-fitting noise removal, and depth raster mapping; vibration noise is effectively compensated and high-fidelity depth maps are generated. A multi-modal segmentation network for cold-rolled strip surface defects is proposed, in which a dual-stream encoder–dual-branch decoder architecture is adopted, a multi-scale feature analysis module is integrated, and a hybrid supervision strategy of stage-wise output supervision plus dual deep–shallow supervision is introduced. Experimental results indicate that optimal defect segmentation performance is achieved on the cold-rolled strip surface multimodal defect dataset and an inference speed of 68 FPS is attained, thereby validating the feasibility of high-precision online defect detection of cold-rolled strip surfaces under vibration interference.

Key words: strip surface defect detection, 3D point cloud processing, multi-modal segmentation, RGB-D network, deep learning

CLC Number:

TJ102

DU Wangzhe, WANG Tao, ZHANG Feiyu, LIU Yaxing, NIU Xiaomiao, LIU Yuanming, WANG Tao, HUANG Qingxue. Research on Multi-modal Segmentation Method for Surface Defects of Cold-rolled Steel Strips Considering Vibration Effects[J]. Journal of Mechanical Engineering, 2026, 62(10): 146-158.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://www.cjmenet.com.cn/EN/10.3901/JME.260339

http://www.cjmenet.com.cn/EN/Y2026/V62/I10/146

References

[1] 黄庆学. 高品质钢铁板带轧制关键装备与技术研究进展[J]. 机械工程学报，2023，59(20)：34-63. HUANG Qingxue. Research progress on key equipment and technologies for rolling high-quality steel sheets and strips[J]. Journal of Mechanical Engineering，2023，59(20)：34-63.
[2] 袁国，王国栋. 高品质钢铁材料轧制加工新技术研究进展及发展趋势[J]. 轧钢，2022，39(6)：12-28. YUAN Guo，WANG Guodong. Research progress and development trend of new technologies for rolling processing of high-quality steel materials[J]. Steel Rolling，2022，39(6)：12-28.
[3] DU Yongzhao，CHEN Haixin，FU Yuqing，et al. AFF-Net：A strip steel surface defect detection network via adaptive focusing features[J]. IEEE Transactions on Instrumentation and Measurement，2024，73：1-14.
[4] TANG Bo，CHEN Li，SUN Wei，et al. Review of surface defect detection of steel products based on machine vision[J]. IET Image Processing，2023，17(2)：303-322.
[5] WANG Tao，DU Wangzhe，LI Xuwei，et al. DDFNet：Real-time salient object detection with dual-branch decoding fusion for steel plate surface defects[J]. Journal of Iron and Steel Research International，2025，32(8)：2421-2433.
[6] 吴葛，朱宇凡，叶天成. 改进YOLOv10的钢材表面缺陷检测方法[J]. 电子测量技术，2025，48(4)：158-168. WU Ge，ZHU Yufan，YE Tiancheng. Improved detection method for steel surface defects of YOLOv10[J]. Electronic Measurement Technology，2025，48(4)：158-168.
[7] 伍麟，郝鸿宇，宋友. 基于计算机视觉的工业金属表面缺陷检测综述[J]. 自动化学报，2024，50(7)：1261-1283. WU Lin，HAO Hongyu，SONG You. Review of industrial metal surface defect detection based on computer vision[J]. Acta Automatica Sinica，2024，50(7)：1261-1283.
[8] CUI Wenqi，SONG Kechen，JIA Xiujian，et al. An efficient targeted design for real-time defect detection of surface defects[J]. Optics and Lasers in Engineering，2024，178：108174.
[9] CHEN Biao，NIU Tongzhi，YU Wenyong，et al. A-net：An a-shape lightweight neural network for real-time surface defect segmentation[J]. IEEE Transactions on Instrumentation and Measurement，2023，73：1-14.
[10] 罗嘉兴，崔瑞钦，马立东. 基于点云层次分析的不锈钢板表面深度探测[J/OL]. 轧钢，1-13[2026-03-02]. https://link.cnki.net/urlid/11.2466.TF.20250326.1457.002. LUO Jiaxing，CUI Ruiqin，MA Lidong. Surface depth detection of stainless steel plates based on point cloud hierarchical analysis[J/OL]. Steel Rolling，1-13[2026- 03-02]. https://link.cnki.net/urlid/11.2466.TF.20250326. 1457.002.
[11] ZHENG Qinbing，ZOU Bin，CHEN Wei，et al. Study on the 3d-printed surface defect detection based on multi-row cyclic scanning method[J]. Measurement，2023，223：113823.
[12] YANG Xiaoping，NI Ping，LI Zhenhua，et al. Dense feature pyramid deep completion network[J]. Electronics，2024，13(17)：3490.
[13] 张介嵩，黄影平，张瑞. 基于CNN的点云图像融合目标检测[J]. 光电工程，2021，48(5)：66-76. ZHANG Jiesong，HUANG Yingping，ZHANG Rui. Point cloud image fusion object detection based on CNN[J]. Opto-electronic Engineering，2021，48(5)：66-76.
[14] SHI Wenxin，GONG Yun，YANG Mengjia，et al. Point cloud depth map and optical image registration based on improved RIFT a lgorithm[C]//2021 IEEE 13th International Conference on Computer Research and Development (ICCRD). IEEE，2021：50-57.
[15] YAN Yunhui，JIA Xiujian，SONG Kechen，et al. Specificity autocorrelation integration network for surface defect detection of no-service rail[J]. Optics and Lasers in Engineering，2024，172：107862.
[16] 贾明达，杨金明，孟维亮，等. 融合点云与图像的环境目标检测研究进展[J]. 中国图象图形学报，2024，29(6)：1765-1784. JIA Mingda，YANG Jinming，MENG Weiliang，et al. Research progress on environmental target detection integrating point cloud and image[J]. Journal of Image and Graphics，2024，29(6)：1765-1784.
[17] HUO Lintao，LIU Ying，YANG Yutu，et al. Research on product surface quality inspection technology based on 3D point cloud[J]. Advances in Mechanical Engineering，2023，15(3)：16878132231159523.
[18] 曾凯，夏梓博，钱俊磊，等. 基于点云多特征融合的钢材表面裂纹特征提取[J]. 激光杂志，2025，46(8)：89-96. ZENG Kai，XIA Zibo，QIAN Junlei，et al. Feature extraction of steel surface cracks based on point cloud multi-feature fusion[J]. Laser Journal，2025，46(8)：89-96.
[19] 吴昆鹏，王少聪，苏成. 基于3D点云的钢管表面缺陷检测系统[J]. 轧钢，2024，41(3)：113-118，126. WU Kunpeng，WANG Shaocong，SU Cheng. Steel pipe surface defect detection system based on 3D point cloud[J]. Steel Rolling，2024，41(3)：113-118，126.
[20] JING Junfeng，WANG Huaqing. Defect segmentation with local embedding in industrial 3D point clouds based on transformer[J]. Measurement Science and Technology，2023，35(3)：035406.
[21] WANG Qihang，WANG Xiaoming，HE Qing，et al. 3D tensor-based point cloud and image fusion for robust detection and measurement of rail surface defects[J]. Automation in Construction，2024，161：105342.
[22] KUMAR S，TIWARI P，CHAUDHURY S B. An optical triangulation method for non-contact profile measurement[C]//2006 IEEE International Conference on Industrial Technology. IEEE，2006：2878-2883.
[23] GUPTA S，GIRSHICK R，ARBELÁEZ P，et al. Learning rich features from RGB-D images for object detection and segmentation[C]//Computer Vision–ECCV 2014：13th European Conference，Zurich，Switzerland，September 6-12，2014，Proceedings，Part VII 13. Springer International Publishing，2014：345-360.
[24] 黄年昌，杨阳，张强，等. 基于深度学习的RGB-D图像显著性目标检测前沿进展[J]. 计算机学报，2025，48(2)：284-316. HUANG Nianchang，YANG Yang，ZHANG Qiang，et al. Frontier progress of salient object detection in RGB-D images based on deep learning[J]. Journal of Computer Science，2025，48(2)：284-316.
[25] WU Yuhuan，LIU Yun，XU Jun，et al. MobileSal: Extremely efficient RGB-D salient object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，44(12)：10261-10269.
[26] HUANG Rui，ZHAO Qingyi，XING Yan，et al. A saliency enhanced feature fusion based multiscale RGB-D salient object detection network[C]//ICASSP 2024-2024 IEEE International Conference on Acoustics，Speech and Signal Processing (ICASSP). IEEE，2024：9356-9360.
[27] NIU Ye，ZHOU Sanping，DONG Yonghao，et al. Bidirectional feature learning network for RGB-D salient object detection[J]. Pattern Recognition，2024，150：110304.
[28] ZHOU Wujie，HONG Jiankang. FHENet：Lightweight feature hierarchical exploration network for real-time rail surface defect inspection in RGB-D images[J]. IEEE Transactions on Instrumentation and Measurement，2023，72：1-8.
[29] SIMONYAN K，ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv Preprint arXiv：1409.1556，2014.
[30] RUSSAKOVSKY O，DENG Jia，SU Hao，et al. Imagenet large scale visual recognition challenge[J]. International Journal of Computer Vision，2015，115：211-252.
[31] WOO S，PARK J，LEE J，et al. Cbam：Convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018：3-19.
[32] CONG Runmin，LIN Qinwei，ZHANG Chen，et al. CIR-Net：Cross-modality interaction and refinement for RGB-D salient object detection[J]. IEEE Transactions on Image Processing，2022，31：6800-6815.