Advances in Research of Computer Vision Structural Health Monitoring and Its Application Prospect in Large Amusement Rides

doi:10.3901/JME.2023.16.167

Abstract

Abstract: In production and life, the structural safety of equipment and facilities is very important to ensure the national economic operation and people’s daily life. Structural health monitoring can effectively improve the safety and stability of equipment and facilities. Structural health monitoring technology based on computer vision measurement has attracted more and more attention with the development of computer vision measurement technology and the improvement of hardware levels. Computer vision measurement has been widely used in the monitoring fields of surface defects, spatial coordinates, displacement and cable tension of structures.However, it is seldom used in structural health monitoring of large amusement rides. In this paper, the overview of computer vision is introduced, the research progresses are summarized and the application prospects are carried out. Firstly, the development history of computer vision is sorted out in chronological order. Then, the hardware and software components of computer vision measurement are summarized, the main features of each component are pointed out. It mainly introduces the application status of computer vision measurement technology in various fields, compares it with traditional measurement methods, and points out the characteristics and shortcomings of various methods. Finally, the application prospect of computer vision measurement technology in structural health monitoring of large amusement rides is put forward. The main research direction and key tasks of computer vision measurement technology in structural health monitoring of large amusement rides are pointed out.

Key words: computer vision, main components, application status, structural health monitoring, large amusement rides

CLC Number:

TH17

YU Zongying, SHEN Gongtian, ZHAO Zhangyan, WU Zhanwen, LIU Yuan. Advances in Research of Computer Vision Structural Health Monitoring and Its Application Prospect in Large Amusement Rides[J]. Journal of Mechanical Engineering, 2023, 59(16): 167-181.

References

[1] KHUC T,CATBAS F. Structural identification using computer vision-based bridge health monitoring[J].Journal of Structural Engineering,2018,144(2):5-12.
[2] FENG D, FENG M Q. Experimental validation of cost-effective vision-based structural health monitoring[J].Mechanical Systems and Signal Processing,2017,88(11):199-211.
[3] 姜绍飞.结构健康监测-智能信息处理及应用[J].工程力学,2009,26(5):487-490.JIANG Shaofei. Structural health monitoring-Intelligent information processing and apllication[J]. Engineering Mechanics,2009,26(5):487-490.
[4] FENG D,FENG M Q. Computer vision for SHM of civil infrastructure:From dynamic response measurement to damage detection:A review[J]. Engineering Structures,2018,156:105-117.
[5] ROBERTS L G. Machine perception of three-dimensional solids[D]. Massachusetts:Massachusetts Institute of Technology,1963.
[6] ROSENFELD A,HUMMEL R A,ZUCKER S W. Scene labeling by relaxation operations[J]. IEEE Transactions on Systems,Man,and Cybernetics,1976(6):420-433.
[7] DAVIS L S. A survey of edge detection techniques[J].Computer Graphics and Image Processing,1975,4(3):248-270.
[8] BAUMGART, BRUCE G. Geometric modeling for computer vision[M]. Stanford:Stanford University,1974.
[9] BAKER M,CHURCHILL J,GILBERT A. The impact of physically attractive models on advertising evaluations[J].Journal of Marketing Research,1977,14(4):538-555.
[10] AGIN G J, THOMAS O. Computer description of curved objects[J]. IEEE Transactions on Computers,1976,25(4):439-449.
[11] NEVATIA,RAMAKANT B,THOMAS O. Description and recognition of curved objects[J]. Artificial Intelligence,1977,8(1):77-98.
[12] MARR D. Representing visual information(A)[J]. Journal of the Optical Society of America(1917-1983),1977,67:1400.
[13] MARR D, VAINA, LUCIA. Representation and recognition of the movements of shapes[J]. Proceedings of the Royal Society of London, Series B:Biological Sciences,1982,214(1197):501-524.
[14] HAGGREN H. Photogrammetric prototype system for real-time engineering applications[C]//Optics in Engineering Measurement. SPIE,1986,599:330-335.
[15] SEITZ,STEVEN M,DYER,et al. Photorealistic scene reconstruction by voxel coloring[J]. International Journal of Computer Vision,1999,35(2):151-173.
[16] KUTULAKOS,KIRIAKOS N,STEVEN M. A theory of shape by space carving[J]. International Journal of Computer Vision,2000,38(3):199-218.
[17] REHG J M,KANADE T. Visual tracking of high dof articulated structures:An application to human hand tracking[C]//Computer Vision-ECCV'94:Third European Conference on Computer Vision Stockholm,Sweden, May 2-61994 Proceedings, Volume II 3.Springer Berlin Heidelberg,1994:35-46.
[18] SHI J. Good features to track[C]//1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition,IEEE,1994:593-600.
[19] LI J C,YUAN B Z. Using vision technique for the bridge deformation detection[C]//ICASSP-88, International Conference on Acoustics,Speech,and Signal Processing,IEEE Computer Society,1988:912-915.
[20] TANG,PEI C,LU S. A video signal processing technique for vibration measurement of a flexible beam[J]. The International Journal of Robotics Research,1992,11(6):579-583.
[21] 张大山.基于高速视觉的结构运动测量算法研究与应用[D].合肥:中国科学技术大学,2017.ZHANG Dashan. Research and applications on vision-based structural motion extraction algorithms[D].Hefei:University of Science and Technology of China,2017.
[22] 雷秀军.高速视觉动态测量系统关键技术与应用研究[D].合肥:中国科学技术大学,2016.LEI Xiujun. Research on key technologies and applications of high speed vision system for dynamic measurement[D]. Hefei:University of Science and Technology of China,2016.
[23] LEE,JONG J,CHO,et al. Evaluation of bridge load carrying capacity based on dynamic displacement measurement using real-time image processing techniques[J]. Steel Struct.,2006,6:377-385.
[24] ZHAO X F,RI K,WANG N N. Experimental verification for cable force estimation using handheld shooting of smartphones[J]. Journal of Sensors, 2017(3):5625396.
[25] HUANG J. Color-Spatial image indexing and applications[D]. Cornell:Cornell Universit,1998.
[26] 徐晨,赵瑞珍,甘小冰.小波分析·应用算法[M].北京:科学出版社,2004.XU Chen,ZHAO Ruizhen,GAN Xiaobing. Wavelet analysis-Application algorithm[M]. Beijing:Science Press,2004.
[27] 周玲,张丽艳,郑建冬,等.近景摄影测量中标记点的自动检测[J].应用科学学报,2007. 25(3):288-294.ZHOU Ling,ZHANG Liyan,ZHENG Jiandong,et al.Automated reference point detectionin close range photogrammetry[J]. Journal of Applied Sciences,2007.25(3):288-294.
[28] YE X W,NI Y Q,WAI T T et al. A vision-based system for dynamic displacement measurement of long-span bridges:algorithm and verification[J]. Smart Structures and Systems,2013,12(3-4):363-379.
[29] KHUC T, CATBAS F. Computer vision-based displacement and vibration monitoring without using physical target on structures[J]. Structure and Infrastructure Engineering,2017,13(4):505-516.
[30] DONG C Z,YE X W,JIN T. Identification of structural dynamic characteristics based on machine vision technology[J]. Measurement,2018,126:405-416.
[31] BAQERSAD J,POOZESH P,NIEZRECKI C et al.Photogrammetry and optical methods in structural dynamics-A review[J]. Mechanical Systems and Signal Processing,2017,86:17-34.
[32] MOLINA V,FELIPE S L,LOPEZ A E,et al. High frequency mode shapes characterisation using digital image correlation and phase-based motion magnification[J]. Mechanical Systems and Signal Processing,2018,102:245-261.
[33] WANG Q, FU L, LIU Z Z. Review on camera calibration[C]//2010 Chinese Control and Decision Conference,IEEE,2010:3354-3358.
[34] 李竹良.快速相机标定算法的研究[D].上海:上海交通大学,2013.LI Zhuliang. Research on fast camera calibration algorithm[D]. Shanghai:Shanghai Jiao Tong University,2013.
[35] YUAN Y,QU Z Y. An adaptive algorithm of camera calibration based on single neuron[J]. Journal of Dalian University of Technology,2005,45(6):823-826.
[36] CAI S,LI Q A,QIAO Y F. Camera calibration of attitude measurement system based on BP neural network[J].Journal of Optoelectronics Laser,2007,18(7):832-845.
[37] JI Q, ZHANG Y. Camera calibration with genetic algorithms[J]. IEEE Transactions on Systems,Man,and Cybernetics-Part A:Systems and Humans,2001,31(2):120-130.
[38] SWAIN M J, BALLARD D H. Color indexing[J].International Journal of Computer Vision,1991,7(1):11-32.
[39] STRICKER M A, ORENGO M. Similarity of color images[C]//Storage and Retrieval for Image and Video Databases III. SPiE,1995,2420:381-392.
[40] PASS G,ZABIH R,MILLER J. Comparing images using color coherence vectors[C]//Proceedings of the Fourth ACM International Conference on Multimedia,1997:65-73.
[41] 孙君顶,马媛媛.纹理特征研究综述[J].计算机系统应用,2010,19(6):245-250.SUN Junding, MA Yuanyuan. Summary of teexture feature research[J]. Computer System Application,2010,19(6):245-250.
[42] 刘丽,匡纲要.图像纹理特征提取方法综述[J].中国图象图形学报,2009,14(4):622-635.LIU Li,KUANG Gangyao. Overview of image textural feature extraction methods[J]. Journal of Image and Graphics,2009,14(4):622-635.
[43] SKLANSKY J. Image segmentation and feature extraction[J]. IEEE Transactions on Systems,Man,and Cybernetics,1978,8(4):237-247.
[44] LI X Q,ZHAO Z W,CHENG H D,et al. A fuzzy logic approach to image segmentation[C]//Proceedings of 12th International Conference on Pattern Recognition,IEEE,1994,1:337-341.
[45] 官云兰,张红军,刘向美.点特征提取算法探讨[J].东华理工学院学报,2007,30(1):42-46.GUAN Yunlan, ZHANG Hongjun, LIU Xiangmei.Research on point feature extraction algorithm[J]. Journal of East China Institute of Technology,2007,30(1):42-46.
[46] 戚世贵.基于图像特征点的提取匹配及应用[D].长春:吉林大学,2006.QI Shigui. The research of features extraction,registration and application[D]. Changchun:Jilin University,2006.
[47] 郭迎福,全伟铭,王文韫,等.基于三维振动信息融合的卷积神经网络风力机叶片裂纹诊断方法[J].光学学报,2020,40(22):140-148.GUO Yingfu,QUAN Weiming,WANG Wenyun,et al.Crack diagnosis method of wind turbine blade based on convolution neural network with 3D vibration information fusion[J]. Acta Optica Sinica,2020,40(22):140-148.
[48] CHOI J,WHANGBO T K,KIM C G. A contour tracking method of large motion object using optical flow and active contour model[J]. Multimedia Tools and Applications,2015,74(1):199-210.
[49] 贾迪,朱宁丹,杨宁华,等.图像匹配方法研究综述[J].中国图像图形学报,2019,24(5):677-699.JIA Di,ZHU Ningdan,YANG Ninghua,et al. Image matching methods[J]. Journal of Image and Graphics,2019,24(5):677-699.
[50] OZBEK M,RIXEN D,ERNE O,et al. Feasibility of monitoring large wind turbines using photogrammetry[J].Energy,2010,35(12):4802-4811.
[51] WAHBEH A M,CAFFREY J P,MASRI S F,et al. A vision-based approach for the direct measurement of displacements in vibrating systems[J]. Smart Materials and Structures,2003,12(5):743-785.
[52] 田娟,郑郁正.模板匹配技术在图像识别中的应用[J].传感器与微系统,2008,27(1):112-114.TIAN Juan,ZHENG Yuzheng. Application of template matching technique in image recognition[J]. Transducer and Microsystem Technologies,2008,27(1):112-114.
[53] SUTTON M A,MCNEIL S R,JANG J,et al. Effects of subpixel image restoration on digital correlation error estimates[J]. Optical Engineering,1988,27(10):870-877.
[54] FENG D M,SCARANGELLO T,FENG M Q,et al.Cable tension force estimate using novel noncontact vision-based sensor[J]. Measurement,2017,99:44-52.
[55] 郭宝云.工业零件形状与尺寸高精度视觉测量方法研究[J].测绘学报,2015(4):466-471.GUO Baoyun. Research on high precision methods of visual measurement of the shape and size of the industrial parts[J]. Journal of Geodesy and Geoinformation Science,2015(4):466-471.
[56] 王建中,姜昱明.基于Snake模型的视频对象分割和跟踪算法[J].计算机工程与设计,2005(7):1928-1930.WANG Jianzhong, JIANG Yuming. Video object segmentation and tracking algorithm based on snake model[J]. Computer Engineering and Design,2005(7):1928-1930.
[57] SURESH B,FUNDAKOWSKI R A,LEVITT T S,et al.A real-time automated visual inspection system for hot steel slabs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1983(6):563-572.
[58] BULNES F G,USAMENTIAGA R,GARCIA D F,et al.Vision-based sensor for early detection of periodical defects in web materials[J]. Sensors, 2012, 12(8):10788-10809.
[59] YAZDCHI M,YAZDI M,MAHYARI A G. Steel surface defect detection using texture segmentation based on multifractal dimension[C]//2009 International Conference on Digital Image Processing,IEEE,2009:346-350.
[60] AI Y H,XU K. Surface detection of continuous casting slabs based on curvelet transform and kernel locality preserving projections[J]. Journal of Iron and Steel Research International,2013,20(5):80-86.
[61] CHEN Y J,TSAI J C,HSU Y C. A real-time surface inspection system for precision steel balls based on machine vision[J]. Measurement Science and Technology,2016,27(7):74-86.
[62] ZHANG S,ZHANG Q,GU J,et al. Visual inspection of steel surface defects based on domain adaptation and adaptive convolutional neural network[J]. Mechanical Systems and Signal Processing,2021,153:12-32.
[63] TAO X,WANG Z H,ZHANG Z H,et al. Wire defect recognition of spring-wire socket using multitask convolutional neural networks[J]. IEEE Transactions on Components,Packaging and Manufacturing Technology,2018,8(4):689-698.
[64] BOUKOUVALAS C,KITTIER J,MARIK R,et al. Color grading of randomly textured ceramic tiles using color histograms[J]. IEEE Transactions on Industrial Electronics,1999,46(1):219-226.
[65] 彭向前.产品表面缺陷在线检测方法研究及系统实现[D].武汉:华中科技大学,2008.PENG Xiangqian. Study on methodology of product surface defects online detection and system implementation[D]. Wuhan:Huazhong University of Science and Technology,2008.
[66] JUNG S Y,TSAI T H,CHIU W Y,et al. Defect detection on randomly textured surfaces by convolutional neural networks[C]//2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics(AIM),IEEE,2018:1456-1461.
[67] 程立.架空高压输电线路巡线机器人视觉检障定位的研究[D].武汉:武汉大学,2019.CHEN Li. Research on visual detection and location for inspection robot for overhead high voltage transmission lines[D]. Wuhan:Wuhan University,2019.
[68] VARADHARAJAN S,JOSE S,SHARMA K,et al. Vision for road inspection[C]//IEEE Winter Conference on Applications of Computer Vision. IEEE,2014:115-122.
[69] SHANG LD,YANG Q,WANG J,et al. Detection of rail surface defects based on CNN image recognition and classification[C]//201820th International Conference on Advanced Communication Technology(ICACT). IEEE,2018:45-51.
[70] 苗建伟.渐开线圆柱齿轮齿廓与径向圆跳动的视觉测量技术研究[D].长春:吉林大学,2021.MIAO Jianwei. Research on visual measurement technology for tooth profile and radial circle runout of the involute cylindrical gear[D]. Changchun:Jilin University,2021.
[71] 包昊菁.基于机器视觉的链轮径向跳动测量技术研究[D].长春:吉林大学,2020.BAO Haoqing. Research on measurement technology of sprocket radial runout based on machine vision[D].Changchun:Jilin University,2020.
[72] 刘少平,杨永波,张东升,等.一种改进的单目视觉桥梁挠度非接触测量方法[J].测绘通报,2021(10):98-102.LIU Shaopin,YANG Yongbo,ZHANG Dongsheng,et al.An improved method of the non-contact measuring method of bridge deflections based on monocular vision[J]. Bulletin of Surveying and Mapping,2021(10):98-102.
[73] 刘巍,李肖,李辉,等.基于双目视觉的数控机床动态轮廓误差三维测量方法[J].机械工程学报,2019,55(10):1-9.LIU Wei,LI Xiao,LI Hui,et al. Binocular vision-based three-dimensional contouring error calibration method for NC machine tools[J]. Journal of Mechanical Engineering,2019,55(10):1-9.
[74] 孙春苗.基于计算机视觉的压气机叶片形状尺寸三维重构方法研究[D].哈尔滨:哈尔滨工业大学,2018.SUN Chunmiao. Research on 3D reconstruction method of shape and dimensions of compressor blades based on computer vision[D]. Harbin:Harbin Institute of Technology,2018.
[75] 李华,雷勇,甘创.基于视觉辅助的隧道轮廓监测[J].机械工程学报,2018,54(1):90-98.LI Hua,LEI Yong,GAN Chuang. Tunnel deformation monitoring based on vision assistant[J]. Journal of Mechanical Engineering,2018,54(1):90-98.
[76] ZHAO B, CHEN W, HU J, et al. An innovative methodology for measurement of stress distribution of inflatable membrane structures[J]. Measurement Science and Technology,2015,27(2):025002.
[77] ZHANG X,GONG W,XU X. Magnetic ring multi-defect stereo detection system based on multi-camera vision technology[J]. Sensors,2020,20(2):392.
[78] LIU S,JIN P,LIU J,et al. Accurate measurement method for tube's endpoints based on machine vision[J]. Chinese Journal of Mechanical Engineering,2017,30:152-163.
[79] WAHBEH A M, CAFFREY J P, MASRI S F. A vision-based approach for the direct measurement of displacements in vibrating systems[J]. Smart Materials and Structures,2003,12(5):785-796.
[80] GWASHAVANHU B,OBERHOLSTER A J,HEYNS P S.Rotating blade vibration analysis using photogrammetry and tracking laser Doppler vibrometry[J]. Mechanical Systems and Signal Processing,2016,76:174-186.
[81] 郭杰.基于视觉的振动特征提取算法研究及应用[D].合肥:中国科学技术大学,2015.GUO Jie. Research and application on vision-based vibration feature extraction algorithm[D]. Hefei:University of Science and Technology of China,2015.
[82] WANG W Y, CHEN A H. Target-less approach of vibration measurement with virtual points constructed with cross ratios[J]. Measurement,2020,151:7-23.
[83] JAVED A,LEE H,KIM B,et al. Vibration measurement of a rotating cylindrical structure using subpixel-based edge detection and edge tracking[J]. Mechanical Systems and Signal Processing,2022,166:3-15.
[84] 高欣.基于CCD技术的汽轮发电机组扭振测量方法研究[D].哈尔滨:哈尔滨工业大学,2006.GAO Xin. Study on torsional oscillation measurement of turbin-generator units based on CCD technology[D].Harbin:Harbin Institute of Technology,2006.
[85] WANG X X, GUO J, LU S, et al. A computer-vision-based rotating speed estimation method for motor bearing fault diagnosis[J]. Measurement Science and Technology,2017,28(6):3-12.
[86] ZHONG J F, ZHONG S C, ZHANG Q, et al.Vision-based system for simultaneous monitoring of shaft rotational speed and axial vibration using non-projection composite fringe pattern[J]. Mechanical Systems and Signal Processing,2019,120:765-776.
[87] KIM S W,JEON B G,KIM N S,et al. Vision-based monitoring system for evaluating cable tensile forces on a cable-stayed bridge[J]. Structural Health Monitoring,2013,12(5-6):440-456.
[88] KIM S W, KIM N S. Dynamic characteristics of suspension bridge hanger cables using digital image processing[J]. NDT&E International,2013,59:25-33.
[89] FENG D M,SCARANGELLO T,FENG M Q,et al.Cable tension force estimate using novel noncontact vision-based sensor[J]. Measurement, 2017, 99:44-52.
[90] 沈功田,刘渊,张君娇,等.在役大型游乐设施健康评价方法研究[J].机械工程学报,2020,56(10):1-11.SHEN Gongtian,LIU Yuan,ZHANG Junjiao,et al,Research on health evaluation method for in-service large-scale amusement rides[J]. Journal of Mechanical Engineering,2020,56(10):1-11.
[91] 刘渊,沈功田,赵章焰,等.大型过山车健康管理方法的应用研究[J].机械工程学报,2017,53(24):85-95.LIU Yuan,SHEN Gongtian,ZHAO Zhangyan,et al.Research and application of health management method of large roller coaster[J]. Journal of Mechanical Engineering,2017,53(24):85-95.
[92] SON K,JEON H,PARK J,et al. Vibration displacement measurement technology for cylindrical structures using camera images[J]. Nuclear Engineering and Technology,2015,47(4):488-499.
[93] DEY N. Uneven illumination correction of digital images:a survey of the state-of-the-art[J]. Optik,2019,183:483-495.