基于传感器融合里程计的相机与激光雷达自动重标定方法

doi:10.3901/JME.2021.20.206

摘要/Abstract

摘要： 智能驾驶车辆行驶环境复杂多样，不可避免地导致传感器相对位姿发生变化，此时需要进行重新标定。针对智能驾驶车辆的相机和激光雷达发生漂移后的重标定问题，提出一种基于传感器融合里程计的自动重标定方法。基于点云投影和图像配准原理建立基准点云和观测图像之间的3D-2D点对，利用N点透视投影得到平移尺度不准的相机运动；通过融合估计的激光雷达运动来恢复准确尺度的相机运动，并将基准点云根据相机运动转换到观测位置下，与观测点云通过点云配准求解变换矩阵，使用时域均值滤波得到最终的外参矩阵。基于智能驾驶车辆试验平台进行室内外实车试验。结果表明所提出的基于传感器融合里程计的方法无需设置标定板，不受环境和数据特征限制，能够实现相机和激光雷达的精确重标定，对传感器漂移具有较高的鲁棒性。

关键词: 智能驾驶车辆, 相机和激光雷达, 传感器漂移, 传感器融合里程计, 重标定

Abstract: The driving environment of the intelligent vehicle is complex and diverse, which inevitably leads to the change of the relative pose of the sensors. Aiming to solving the problem of recalibration after the drift of camera or LiDAR equipped on the intelligent vehicle, the automatic recalibration method using sensor fusion odometry is proposed. First, based on the principle of point cloud projection and image registration, the 3D-2D point correspondences between the reference point cloud and the observed image are established, and the camera motion with inaccurate translation scale is estimated by the Perspective-n-Point method; Then, the accurate camera motion is recovered by fusing the estimated LiDAR motion, the reference point cloud is transformed to the observation position using the camera motion, and registrated with the observation point cloud, the final extrinsic parameter is obtained using the average filtering method. Finally, the real vehicle experiments are implemented in indoor and outdoor scenario on an intelligent vehicle. The results show that the proposed sensor fusion odometry based method does not need to set up the calibration board, and is unrestricted with the environment and data features. It is able to achieve accurate recalibration of camera and LiDAR, and has high robustness to sensor drift.

Key words: intelligent vehicle, camera and LiDAR, sensor drift, sensor fusion odometry, recalibration

中图分类号:

U463

彭湃, 耿可可, 殷国栋, 庄伟超, 刘帅鹏, 徐利伟. 基于传感器融合里程计的相机与激光雷达自动重标定方法[J]. 机械工程学报, 2021, 57(20): 206-214.

PENG Pai, GENG Keke, YIN Guodong, ZHUANG Weichao, LIU Shuaipeng, XU Liwei. Automatic Recalibration of Camera and LiDAR Using Sensor Fusion Odometry[J]. Journal of Mechanical Engineering, 2021, 57(20): 206-214.

参考文献

[1] GU S,ZHANG Y,TANG J,et al. Integrating dense lidar-camera road detection maps by a multi-modal crf model[J]. IEEE Transactions on Vehicular Technology,2019,68(12):11635-11645.
[2] BASSO F,MENEGATTI E,PRETTO A. Robust intrinsic and extrinsic calibration of rgb-d cameras[J]. IEEE Transactions on Robotics,2018,34(5):1315-1332.
[3] WALLACE A M,HALIMI A,BULLER G S. Full waveform lidar for adverse weather conditions[J]. IEEE Transactions on Vehicular Technology,2020,69(7):7064-7077.
[4] CHEN X,MA H,WAN J,et al. Multi-view 3d object detection network for autonomous driving[C]//Institute of Electrical and Electronics Engineers. IEEE Conference on Computer Vision and Pattern Recognition,July 21-26,2017,Honolulu,Hawaii. New York:IEEE,2017:6526-6534
[5] RANGESH A,TRIVEDI M M. No blind spots:full-surround multi-object tracking for autonomous vehicles using cameras and lidars[J]. IEEE Transactions on Intelligent Vehicles,2019,4(4):588-599.
[6] 孙殿柱,沈江华,李延瑞,等. 序列图像约束的点云初始配准方法[J]. 机械工程学报,2020,56(9):229-236. SUN Dianzhu,SHEN Jianghua,LI Yanrui,et al. Initial registration method of point cloud based on sequence images[J]. Journal of Mechanical Engineering,2020,56(9):229-236.
[7] ZHEN W,HU Y,LIU J,et al. A joint optimization approach of lidar-camera fusion for accurate dense 3-d reconstructions[J]. IEEE Robotics and Automation Letters,2019,4(4):3585-3592.
[8] 丁雅斌,梅江平,张文昌,等. 基于单目视觉的并联机器人末端位姿检测[J]. 机械工程学报,2014,50(21):174-179. DING Yabin,MEI Jiangping,ZHANG Wenchang,et al. Position and orientation measurement of parallel robot based on monocular vision[J]. Journal of Mechanicd Engineering,2014,50(21):174-179.
[9] 乔贵方,孙大林,宋光明,等. 串联机器人标定系统的坐标系快速转换方法[J]. 机械工程学报,2020,56(14):1-8. QIAO Guifang,SUN Dalin,SONG Guangming,et al. A rapid coordinate transformation method for serial robot calibration system[J]. Journal of Mechanical Engineering,2020,56(14):1-8.
[10] 张铁,叶景杨,刘晓刚. 面向机器人砂带打磨的加权手眼标定算法[J]. 机械工程学报,2018,54(17):142-148. ZHANG Tie,YE Jingyang,LIU Xiaogang. Weighted hand-eye calibration algorithm for robot grinding[J]. Journal of Mechanical Engineering,2018,54(17):142-148.
[11] 李兵,傅卫平,王雯,等. 一种基于EIH的装配机器人标定方法[J]. 机械工程学报,2018,54(7):38-44. LI Bing,FU Weiping,WANG Wen,et al. Eih based calibration method for assembly robot[J]. Journal of Mechanical Engineering,2018,54(7):38-44.
[12] 薛培林,吴愿,殷国栋,等. 基于信息融合的城市自主车辆实时目标识别[J]. 机械工程学报,2020,56(12):165-173. XUE Peilin,WU Yuan,YIN Guodong,et al. Real-time target recognition for urban autonomous vehicles based on information fusion[J]. Journal of Mechanical Engineering,2020,56(12):165-173.
[13] WANG W,SAKURADA K,KAWAGUCHI N. Reflectance intensity assisted automatic and accurate extrinsic calibration of 3d lidar and panoramic camera using a printed chessboard[J]. Remote Sensing,2017,9(8):851.
[14] AN Y,LI B,HU H,et al. Building an omnidirectional 3d color laser ranging system through a novel calibration method[J]. IEEE Transactions on Industrial Electronics,2019,66(11):8821-8831.
[15] ITAMI F,YAMAZAKI T. An improved method for the calibration of a 2d lidar with respect to a camera by using a checkerboard target[J]. IEEE Sensors Journal,2020,20(14):7906-7917.
[16] ZHANG Q,PLESS R. Extrinsic calibration of a camera and laser range finder (improves camera calibration)[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems,September 28-Octorber 2,2004,Sendai,Japan. New York:IEEE,2004:2301-2306.
[17] GEIGER A,MOOSMANN F,CAR Ö,et al. Automatic camera and range sensor calibration using a single shot[C]//IEEE International Conference on Robotics and Automation,May 14-18,2012,Saint Paul,MN,USA. New York:IEEE,2012:3936-3943.
[18] 康国华,张琪,张晗,等. 基于点云中心的激光雷达与相机联合标定方法研究[J]. 仪器仪表学报,2019,40(12):121-129. KANG Guohua,ZHANG Qi,ZHANG Han,et al. Joint calibration of camera and lidar based on point cloud center[J]. Chinese Journal of Scientific Instrument,2019,40(12):121-129.
[19] CHAI Z,SUN Y,XIONG Z. A novel method for lidar camera calibration by plane fitting[C]//2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics,July 9-12,2018,Auckland,New Zealand. New York:ASME,2018:286-291.
[20] KANG J,DOH N L. Full-dof calibration of a rotating 2-d lidar with a simple plane measurement[J]. IEEE Transactions on Robotics,2016,32(5):1245-1263.
[21] PARK Y,Yun S,Won C,et al. Calibration between color camera and 3d lidar instruments with a polygonal planar board[J]. Sensors,2014,14(3):5333-53.
[22] XIE S,YANG D,JIANG K,et al. Pixels and 3-d points alignment method for the fusion of camera and lidar data[J]. IEEE Transactions on Instrumentation and Measurement,2019,68(10):3661-3676.
[23] CAI H,PANG W,CHEN X,et al. A novel calibration board and experiments for 3d lidar and camera calibration[J]. Sensors,2020,20(4):1130.
[24] 许涵博. 面向自动驾驶系统的相机-激光雷达的自动标定[D]. 哈尔滨:哈尔滨工业大学,2019. XU Hanbo. Online intelligent calibration of cameras and lidars for autonomous driving systems[D]. Harbin:Harbin Institute of Technology,2019.
[25] GONG X,LIN Y,LIU J. 3d lidar-camera extrinsic calibration using an arbitrary trihedron[J]. Sensors,2013,13(2):1902-1918.
[26] CUI T,JI S,SHAN J,et al. Line-based registration of panoramic images and lidar point clouds for mobile mapping[J]. Sensors,2017,17(1):70.
[27] BROWN M,WINDRIDGE D,GUILLEMAUT J Y. A family of globally optimal branch-and-bound algorithms for 2d-3d correspondence-free registration[J]. Pattern Recognition,2019,93:36-54.
[28] PANDEY G,MCBRIGE J R,SAVARESE S,et al. Automatic extrinsic calibration of vision and lidar by maximizing mutual information[J]. Journal of Field Robotics,2015,32(5):696-722.
[29] HUANG X,ZHANG J,FAN L,et al. A systematic approach for cross-source point cloud registration by preserving macro and micro structures[J]. IEEE Transactions on Image Processing,2017,26(7):3261-3276.
[30] YANG J,QUAN S,WANG P,et al. Evaluating local geometric feature representations for 3D rigid data matching[J]. IEEE Transactions on Image Processing,2019,29:2522-2535.
[31] PENATE-SANCHEZ A,ANDRADE-CETTO J,MORENO-NOGUER F. Exhaustive linearization for robust camera pose and focal length estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(10):2387-2400.
[32] HADFIELD S,LEBEDA K,BOWDEN R. Hard-pnp:pnp optimization using a hybrid approximate representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,41(3):768-774.