Overview of Key Technologies for Measurement Robots in Intelligent Manufacturing

doi:10.3901/JME.2024.16.001

Abstract

Abstract: Complex curved components are the core elements of high-end equipment in fields such as aerospace and marine vessels, and their measurement accuracy plays an irreplaceable role in ensuring the quality of high-end equipment manufacturing. To overcome the limitations of traditional manual and specialized manufacturing methods, vision-guided robotic systems provide a new approach for the high-end and intelligent processing of complex curved components, gradually becoming a research hot spot in the field of robotic intelligent manufacturing. Focusing on the 3D measurement methods of robots, this review first summarizes the characteristics of measurement schemes in different manufacturing scenarios according to sensor types and application scenarios, so as to help researchers quickly and comprehensively understand this field. Then, according to the measurement process, key core technologies are categorized as system calibration, measurement planning, point cloud fusion, feature recognition, etc. The major research achievements in various categories over the past decade are reviewed, and the existing research limitations are analyzed. Finally, the technical challenges faced by robotic measurement are summarized, and future development trends are discussed from the perspectives of application scenarios, measurement requirements, measurement methods, etc.

Key words: intelligent manufacturing, robotic measurement, system calibration, scanning planning, feature recognition

CLC Number:

TP242

WANG Yaonan, XIE He, DENG Jingdan, MAO Jianxu, LI Wenlong, ZHANG Hui. Overview of Key Technologies for Measurement Robots in Intelligent Manufacturing[J]. Journal of Mechanical Engineering, 2024, 60(16): 1-18.

References

[1] TANG Y,WANG Y,TAN H,et al. Viewpoint planning of robotic measurement system for free-form surfaces based on visibility cone space explorer[J]. IEEE Transactions on Automation Science and Engineering,2023:1-15.
[2] TANG Y,WANG Y,TAN H,et al. A digital twin-based intelligent robotic measurement system for freeform surface parts[J]. IEEE Transactions on Instrumentation and Measurement,2023,72(6):7003013.
[3] ERNST F,RICHTER L,MATTHÄUS L,et al. Non-orthogonal tool/flange and robot/world calibration[J]. The International Journal of Medical Robotics and Computer Assisted Surgery,2012,8(4):407-420.
[4] REN H,KAZANZIDES P. A paired-orientation alignment problem in a hybrid tracking system for computer assisted surgery[J]. Journal of Intelligent Robotic Systems,2011,63(2):151-161.
[5] LI A,WANG L,WU D. Simultaneous robot-world and hand-eye calibration using dual-quaternions and Kronecker product[J]. International Journal of Physical Sciences,2010,5(10):1530-1536.
[6] HA J. Probabilistic framework for hand-eye and robot-world calibration AX=YB[J]. IEEE Transactions on Robotics,2023,39(2):1196-1211.
[7] WANG G,LI W L,JIANG C,et al. Simultaneous calibration of multicoordinates for a dual-robot system by solving the AXB=YCZ problem[J]. IEEE Transactions on Robotics,2021,37(4):1172-1185.
[8] WU L,WANG J,QI L,et al. Simultaneous hand-eye,tool-flange,and robot-robot calibration for comanipulation by solving the AXB=YCZ problem[J]. IEEE Transactions on Robotics,2016,32(2):413-428.
[9] JIANG J F,LUO X,XU S J,et al. Hand-eye calibration of eod robot by solving the AXB=YCZD problem[J]. IEEE Access,2022,10:3415-3429.
[10] TSAI R Y,LENZ R K. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration[J]. IEEE Transactions on Robotics and Automation,1989,5(3):345-358.
[11] XU X,ZHU D,ZHANG H,et al. TCP-based calibration in robot-assisted belt grinding of aero-engine blades using scanner measurements[J]. The International Journal of Advanced Manufacturing Technology,2017,90(1-4):635-647.
[12] ULRICH M,STEGER C. Hand-eye calibration of SCARA robots using dual quaternions[J]. Pattern Recognition and Image Analysis,2016,26(1):231-239.
[13] BRIESE D,NIEBLER C,ROSE G. Hand-eye calibration using dual quaternions in medical environment[J]. Medical Imaging,2014,9036:1-6.
[14] HA J,KANG D,PARK F. A stochastic global optimization algorithm for the two-frame sensor calibration problem[J]. IEEE Transactions on Industrial Electronics,2016,63(4):2434-2446.
[15] LI J,CHEN M,JIN X,et al. Calibration of a multiple axes 3-D laser scanning system consisting of robot,portable laser scanner and turntable[J]. Optik-International Journal for Light and Electron Optics,2011,122(4):324-329.
[16] LISKA J,VANICEK O,CHALUS M. Hand-eye calibration of a laser profile scanner in robotic welding[C]//International Conference on Advanced Intelligent Mechatronics. Auckland,New Zealand:IEEE,2018:316-321.
[17] CHEN W,DU J,XIONG W,et al. A noise-tolerant algorithm for robot-sensor calibration using a planar disk of arbitrary 3-D orientation[J]. IEEE Transactions on Automation Science and Engineering,2018,15(1):251-263.
[18] SHI Q,XI N,ZHANG C. Develop a robot-aided area sensing system for 3D shape inspection[J]. Journal of Manufacturing Science and Engineering,2010,132(1):014502.
[19] YU C Y,XI J T. Simultaneous and on-line calibration of a robot-based inspecting system[J]. Robotics and Computer-Integrated Manufacturing,2018(49):49349-49360.
[20] LU B,LI B,DOU Q,et al. A unified monocular camera-based and pattern-free hand-to-eye calibration algorithm for surgical robots with RCM constraints[J]. IEEE-ASME Trans. Mechatron.,2022,27(6):5124-5135.
[21] OZGUNER O,SHKURTI T,HUANG S Q,et al. Camera-robot calibration for the da vinci robotic surgery system[J]. IEEE Transactions on Automation Science and Engineering,2020,17(4):2154-2161.
[22] HU J S,CHANG Y J. Automatic calibration of hand-eye-workspace and camera using hand-mounted line laser[J]. IEEE/ASME Transactions on Mechatronics,2013,18(6):1778-1786.
[23] LEMBONO T S,SUÁREZ-RUIZ F,PHAM Q C,et al. Scalar:simultaneous calibration of 2D laser and robot kinematic parameters using planarity and distance constraints[J]. IEEE Transactions on Automation Science and Engineering,2019,16(4):1971-1979.
[24] SHIBIN Y,YONGJIE R,JIGUI Z,et al. A vision-based self-calibration method for robotic visual inspection systems[J]. Sensors,2013,13(12):16565-16582.
[25] CAJAL C J R,MANUFACTURING C I. A crenellated-target-based calibration method for laser triangulation sensors integration in articulated measurement arms[J]. Robotics and Computer-Integrated Manufacturing,2011,27(2):282-291.
[26] LIU X J,MADHUSUDANAN H,CHEN W Y,et al. Fast eye-in-hand 3-D scanner-robot calibration for low stitching errors[J]. IEEE Transactions on Industrial Electronics,2021,68(9):8422-8432.
[27] XIE H,LI W L,LIU H. General geometry calibration using arbitrary free-form surface in a vision-based robot system[J]. IEEE Transactions on Industrial Electronics,2022,69(6):5994-6003.
[28] YIN S,REN Y,YIN G,et al. Development and calibration of an integrated 3D scanning system for high-accuracy large-scale metrology[J]. Measurement,2014,54(6):65-76.
[29] 王晨学. 基于视觉辅助定位的机器人自标定技术研究[D]. 无锡:江南大学,2018. WANG Chenxue. Research on robot self-calibration technology based on vision-aided positioning[D]. Wuxi:Jiangnan University,2018.
[30] MAYBANK S J,FAUGERAS O D. A theory of self-calibration of a moving camera[J]. Int. J. Comput. Vision,1992,8(2):123-151.
[31] 邓子贤. 野外大视场立体视觉空间坐标测系统自标定方法研究[D]. 天津:天津大学,2017. DENG Zixian. Research on self-calibration method of stereo visual spatial coordinate measurement system with large field of view in the field[D]. Tianjin:Tianjin University,2017.
[32] STEPANOVA K,PAJDLA T,HOFFMANN M. Robot self-calibration using multiple kinematic chains-a simulation study on the icub humanoid robot[J]. IEEE Robotics and Automation Letters,2019,4(2):1900-1907.
[33] TING Y Y,LI T,FU X,et al. Zooming-free hand-eye self-calibration for nanorobotic manipulation inside SEM[J]. IEEE Transaction Nanotechnol,2023,22291- 22300.
[34] ZHU Q D,XIE X R,LI C,et al. Kinematic self-calibration method for dual-manipulators based on optical axis constraint[J]. IEEE Access,2019,7:7768-7782.
[35] 李宝全,方勇纯,张雪波. 基于纯旋转运动的摄像机统一自标定方法[J]. 光学学报,2013,33(11):154-161. LI Baoquan,FANG Yongchun,ZHANG Xuebo. Unified self-calibration method for camera based on pure rotational motion[J]. Acta Optica Sinica,2013,33(11):154-161.
[36] SUN J,WANG P,QIN Z,et al. Effective self-calibration for camera parameters and hand-eye geometry based on two feature points motions[J]. IEEE/CAA Journal of Automatica Sinica,2017,4(2):370-380.
[37] 许海霞,王耀南,万琴,等. 一种机器人手眼关系自标定方法[J]. 机器人,2008,30(4):373-378. XU Haixia,WANG Yaonan,WAN Qin,et al. A self-calibration method for hand-eye relationship in robots[J]. Robot,2008,30(4):373-378.
[38] DORNHEGE C,KLEINER E,KOLLING A. Coverage search in 3D[C]//2013 IEEE International Symposium on Safety,Security,and Rescue Robotics(SSRR). IEEE,2013:1-8.
[39] BIRCHER A,KAMEL M,ALEXIS K,et al. Three-dimensional coverage path planning via viewpoint resampling and tour optimization for aerial robots[J]. Auton Robot,2015,40(6):1-20.
[40] WEI J,KENJI S. Model-based view planning for building inspection and surveillance using voxel dilation,medial objects,and random-key genetic algorithm[J]. Journal of Computational Design and Engineering,2018,5(3):337-347.
[41] REN H W,QURESHI A H. Robot active neural sensing and planning in unknown cluttered environments[J]. IEEE Transactions on Robotics,2023,39(4):13.
[42] TORABI L,GUPTA K. An autonomous six-DOF eye-in-hand system for in situ 3D object modeling[J]. The International Journal of Robotics Research,2012,31(1):82-100.
[43] MA H X,ZOU W,SUN S Y,et al. FOV constraint region analysis and path planning for mobile robot with observability to multiple feature points[J]. Int. J. Control Autom. Syst.,2021,19(11):3785-3800.
[44] SANCHEZ P,CASADO R,BERMUDEZ A. Real-time collision-free navigation of multiple UAVs based on bounding boxes[J]. Electronics,2020,9(10):19.
[45] MONICA R,ALEOTTI J. Contour-based next-best view planning from point cloud segmentation of unknown objects[J]. Autonomous Robots,2017,42:1-16.
[46] VASQUEZ-GOMEZ J I,SUCAR L E,MURRIETA-CID R. View/state planning for three-dimensional object reconstruction under uncertainty[J]. Autonomous Robots,2015,41(1):1-21.
[47] IRVING V,ENRIQUE S L,RAFAEL M C,et al. Volumetric next-best-view planning for 3D object reconstruction with positioning error[J]. International Journal of Advanced Robotic Systems,2014,11(159):1-13.
[48] ZHAO W Z,LIU Y H,LI Y Z,et al. Multi-robot coverage path planning for dimensional inspection of large free-form surfaces based on hierarchical optimization[J]. International Journal of Advanced Manufacturing Technology,2023,127(11-12):5471-5486.
[49] LAURI M,PAJARINEN J,PETERS J,et al. Multi-sensor next-best-view planning as matroid-constrained submodular maximization[J]. IEEE Robot and Automation Letters,2020,5(4):5323-5330.
[50] LIU Y H,ZHAO W Z,LIU H P,et al. Coverage path planning for robotic quality inspection with control on measurement uncertainty[J]. IEEE-ASME Transactions on Mechatronics,2022,27(5):3482-3493.
[51] YAN L,WEI P C,XIE H,et al. A new outlier removal strategy based on reliability of correspondence graph for fast point cloud registration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(7):7986-8002.
[52] BESL P,MCKAY N. Method for registration of 3-D shapes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,1992,14(2):239-256.
[53] LU F,CHEN G,LIU Y L,et al. Sparse-to-dense matching network for large-scale lidar point cloud registration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(9):11270-11282.
[54] MUNOZ F,COMPORT A I. Global point-to-hyperplane ICP:local and global pose estimation by fusing color and depth[C]//International Conference on Multisensor Fusion & Integration for Intelligent Systems(MFI). Daegu,Korea(South):IEEE,2017:22-27.
[55] POTTMANN H,HUANG Q,YANG Y,et al. Geometry and convergence analysis of algorithms for registration of 3D shapes[J]. International Journal of Computer Vision,2006,67(3):277-296.
[56] KORN M,HOLZKOTHEN M,PAULI J. Color supported generalized-ICP[C]//International Conference on Computer Vision Theory & Applications. Lisbon,Portugal:VISAPP,2014:592-599.
[57] XIE H,WANG Y N,ZHANG H,et al. Iterative fine point-set matching of a free-form surface based on the point-to-sphere distance[J]. IEEE-ASME Trans. Mechatron.,2022,27(6):5668-5678.
[58] XIE H,LI W L,YIN Z P,et al. Variance-minimization ierative matching method for free-form surfaces-part I:Theory and method[J]. IEEE Transactions on Automation Science and Engineering,2019,16(3):1181-1191.
[59] YANG J,LI H,JIA Y. Go-ICP:Solving 3D registration efficiently and globally optimally[C]//International Conference on Computer Vision. Sydney,NSW,Australia:IEEE,2013:1457-1464.
[60] MAGNUSSON M,LILIENTHAL A,DUCKETT T. Scan registration for autonomous mining vehicles using 3D-NDT[J]. Journal of Field Robotics,2010,24(1):803-827.
[61] SHEN D,LIN Y X,REN Z,et al. Robust and efficient GMM-based free-form parts registration via bi-directional distance[J]. Neurocomputing,2019,360:279-293.
[62] FU K X,LI Z H,XU M Y,et al. Dual-branch deep point cloud registration framework for unconstrained rotation[J]. IEEE Trans. Ind. Inform,2023,19(7):7851-7861.
[63] YEW Z J,LEE G H. RPM-Net:Robust point matching using learned features[C]//Conference on Computer Vision and Pattern Recognition. Seattle,WA,USA:IEEE,2020:11821-11830.
[64] LI J,ZHANG C,XU Z,et al. Iterative distance-aware similarity matrix convolution with mutual-supervised point elimination for efficient point cloud registration[C]//European Conference on Computer Vision. Glasgow,United Kingdom:ECCV,2020:378-394.
[65] WANG Y,SOLOMON J M. Deep closest point:Learning representations for point cloud registration[C]//International Conference on Computer Vision. Seoul,Korea(South):IEEE,2019:3522-3531.
[66] AOKI Y,GOFORTH H,SRIVATSAN R A,et al. Pointnetlk:Robust & efficient point cloud registration using pointnet[C]//Conference on Computer Vision and Pattern Recognition. Long Beach,CA,USA:IEEE,2019:7156-7165.
[67] WANG Y J,YAN C G,FENG Y T,et al. Storm:Structure-based overlap matching for partial point cloud registration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(1):1135-1149.
[68] GOJCIC Z,ZHOU C,WEGNER J D,et al. Learning multiview 3D point cloud registration[C]//Conference on Computer Vision and Pattern Recognition. Seattle,WA,USA:CVPR,2020:1-11.
[69] CUI Z,PING T. Global structure-from-motion by similarity averaging[C]//International Conference on Computer Vision. Santiago,Chile:IEEE,2015:864-872.
[70] LI D,CHEN F. DeepMapping:Unsupervised map estimation from multiple point clouds[C]//Conference on Computer Vision and Pattern Recognition. Long Beach,CA,USA:IEEE,2020:8642-8651.
[71] BERGEVIN R,SOUCY M,GAGNON H,et al. Towards a general multi-view registration technique[J]. Transactions on Pattern Analysis and Machine Intelligence,1996,18(5):540-547.
[72] 徐思雨,祝继华,田智强,等. 逐步求精的多视角点云配准方法[J]. 自动化学报,2019,45(8):1486-1494. XU Siyu,ZHU Jihua,TIAN Zhiqiang,et al. Multi-view point cloud registration method for gradual refinement[J]. Acta Automatica Sinica,2019,45(8):1486-1494.
[73] GOVINDU V M,POOJA A. On averaging multiview relations for 3d scan registration[J]. IEEE Transactions on Image Processing,2014,23(3):1289-1302.
[74] ZHU J,JIANG Z,EVANGELIDIS G D,et al. Efficient registration of multi-view point sets by K-means clustering[J]. Information Sciences,2019,488:205-218.
[75] EVANGELIDIS G D,HORAUD R. Joint alignment of multiple point sets with batch and incremental expectation-maximization[J]. IEEE transactions on Pattern Analysis and Machine Intelligence,2016,40(6):1397-1410.
[76] LEIBE B,MATAS J,SEBE N,et al. Global registration of 3D point sets via LRS decomposition[C]//European Conference on Computer Vision. Amsterdam,Netherlands:ECCV,2016:489-504.
[77] 马洁莹,田暄,翟庆,等. 基于点到面度量的多视角点云配准方法[J]. 西安交通大学学报,2022,56(6):120-132. MA Jieying,TIAN Xuan,ZHAI Qing,et al. Multi-view point cloud registration method based on point-to-area measurement[J]. Journal of Xi'an Jiaotong University,2022,56(6):120-132.
[78] WEI M Q,WEI Z Y,ZHOU H R,et al. AGConv:Adaptive graph convolution on 3D point clouds[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(8):9374-9392.
[79] XIE Q,LU D N,DU K P,et al. Aircraft skin rivet detection based on 3D point cloud via multiple structures fitting[J]. Computer-Aided Design,2020,120:1-12.
[80] LONG K,XIE Q,LU D N,et al. Aircraft skin gap and flush measurement based on seam region extraction from 3D point cloud[J]. Measurement,2021,176:1-11.
[81] FAN J J,MA L Q,SUN A B,et al. An approach for extracting curve profiles based on scanned point cloud[J]. Measurement,2020,149:1-7.
[82] 柯映林,陈曦. 点云数据的几何属性分析及区域分割[J]. 机械工程学报,2006,42(8):7-15. KE Yinglin,CHEN Xi. Geometric attribute analysis and region segmentation of point cloud data[J]. Journal of Mechanical Engineering,2006,42(8):7-15.
[83] MAHDI S,SHERVIN D,BENJAMIN B,et al. Graphite:Graph-induced feature extraction for point cloud registration[C]//2020 International Conference on 3D Vision(3DV). Fukuoka,Japan:IEEE,2020:241-251.
[84] GUMHOLD S,WANG X,MACLEOD R. Feature extraction from point clouds[C]//Proceedings of 10th International Meshing Roundtable Conference. Newport Beach,CA,USA:Computer Science,Engineering,2001:1-10.
[85] KUÇAK R A. The feature extraction from point clouds using geometric features and RANSAC algorithm[J]. Advanced LiDAR,2022,2(1):15-20.
[86] ALSHAWABKEH Y. Linear feature extraction from point cloud using color information[J]. Herit Sci,2020,8(28):1-13.
[87] ZHAO M F,CAO L B,SONG T,et al. Independent method for selecting radius of fpfh neighborhood in 3D point cloud registration[J]. Laser Optoelectron. Prog.,2021,58(6):9.
[88] CHARLES R Q,HAO S,KAICHUN M,et al. PointNet:Deep learning on point sets for 3D classification and segmentatio[C]//IEEE Conference on Computer Vision and Pattern Recognition. Honolulu,HI,USA:IEEE,2017:77-85.
[89] QI R,YI L,SU H. PointNet++:Deep hierarchical feature learning on point sets in a metric space[C]//In Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY,USA:Curran Associates Inc,2017:5105-5114.
[90] ZHAO H,JIANG L,FU W. Pointweb:Enhancing local neighborhood features for point cloud processing[C]//Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach,CA,USA:IEEE,2019:5560-5568.
[91] LI Y,BU R,SUN M. PointCNN:Convolution on x-transformed points[C]//The Annual Conference on Neural Information Processing Systems. Montréal,Canada:NIPS,2018:828-838.
[92] SARODE V,LI X,GOFORTH H. PCRNet:point cloud registration network using pointnet encoding[J]. Computer Vision and Pattern Recognition,2019,ArXiv,abs/1908.07906.
[93] WANG Y,SOLOMON J. Deep closest point:Learning representations for point cloud registration[C]//International Conference on Computer Vision(ICCV). Seoul,Korea(South):IEEE,2019:3523-3532.
[94] WANG Y,SOLOMON J. Prnet:Self-supervised learning for partial-to-partial registration[C]//Conference on Neural Information Processing Systems. Red Hook,NY,USA:Curran Associates Inc,2019:8814-8826.
[95] CHOY C,DONG W,KOLTUN V. Deep global registration[C]//Conference on Computer Vision and Pattern Recognition. Seattle,WA,USA:IEEE,2020:2511-2520.
[96] HUANG S,GOJCIC Z,USVYATSOV M. Predator registration of 3D point cloudswith low overlap[C]//Proceedings of the IEEE/CVF Conforence on Computer Vision and Pattern Recognition,2021:4267-4276.
[97] EBBESEN C,FROEMKE R. Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography[J]. Nature Communications,2022,13(1):1-21.
[98] LEE Y,DO W,YOON H. Visual-inertial hand motion tracking with robustness against occlusion,interference,and contact[J]. Science Robotics,2021,6(58):eabe1315.
[99] DESINGH K,LU S,OPIPARI A. Efficient nonparametric belief propagation for pose estimation and manipulation of articulated objects[J]. Science Robotics,2019,4(30):eaaw4523.
[100] XIE H,LI W L,YIN Z P,et al. Variance-minimization iterative matching method for free-form surfaces-part I:Theory and method[J]. IEEE Transactions on Automation Science and Engineering,2019,16(3):1181-1191.
[101] 谢核. 机器人加工几何误差建模及工程应用[D]. 武汉:华中科技大学,2019. XIE He. Modeling and engineering application of geometric error of robot machining[D]. Wuhan:Huazhong University of Science and Technology,2019.
[102] 王刚. 航空蒙皮机器人铣削加工位姿参数标定与光顺路径规划[D]. 武汉:华中科技大学,2023. WANG Gang. Calibration of pose parameters and smooth path planning for milling processing of aviation skin robot[D]. Wuhan:Huazhong University of Science and Technology,2023.
[103] WU Z J,WANG Y N,XIE H,et al. A systematic point cloud edge detection framework for automatic aircraft skin milling[J]. IEEE Transactions on Industrial Informatics,2024,20(1):560-572.