Object Detection and Tracking Based on Image and Point Clouds Instance Matching for Intelligent Vehicles

doi:10.3901/JME.2024.22.302

Abstract

Abstract: In the environment perception task of intelligent vehicles, in order to combine the rich semantic information in the camera image with the accurate spatial information in the lidar point clouds, a fusion detection method based on image and point clouds instance matching is proposed. To achieve the fusion detection, the instance masks of the targets in the image are predicted by the instance segmentation network, the point clouds are projected to the image plane through perspective projection transformation, the point clouds belonging to the target are extracted according to the instance mask of each target, and then the clustering algorithm is used to remove the noise, and the convex hull approximating algorithm is used to fit the 3D bounding box of the target. Based on the fusion detection method, a gate is designed to realize multi-target data association and management, and the Kalman filter is used to track the target and estimate the motion state of the target. The experimental results show that the method can effectively fuse the information from image data and point clouds data, accurately and quickly fit the position, size, and direction of the target and estimate the speed of the target, and show robustness in different experimental scenarios.

Key words: intelligent vehicle, object detection, object tracking, sensor fusion, instance segmentation, perspective projection

CLC Number:

TG156

LI Shangjie, YIN Guodong, GENG Keke, LIU Shuaipeng. Object Detection and Tracking Based on Image and Point Clouds Instance Matching for Intelligent Vehicles[J]. Journal of Mechanical Engineering, 2024, 60(22): 302-310.

References

[1] PENG P，GENG K，YIN G，et al. Adaptive multi-modal fusion instance segmentation for CAEVs in complex conditions：Dataset，framework and verifications[J]. Chinese Journal of Mechanical Engineering，2021，34(1)：1-11.
[2] LIU L，OUYANG W，WANG X，et al. Deep learning for generic object detection：A survey[J]. International Journal of Computer Vision，2020，128(2)：261-318.
[3] REN S，HE K，GIRSHICK R，et al. Faster R-CNN：Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39(6)：1137-1149.
[4] REDMON J，DIVVALA S，GIRSHICK R，et al. You only look once：Unified，real-time object detection[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：779-788.
[5] LIU W，ANGUELOV D，ERHAN D，et al. SSD：Single shot multibox detector[C]// European Conference on Computer Vision，2016：21-37.
[6] HE K，GKIOXARI G，DOLLÁR P，et al. Mask R-CNN[C]// Proceedings of the IEEE International Conference on Computer Vision，2017：2961-2969.
[7] LI Y，QI H，DAI J，et al. Fully convolutional instance- aware semantic segmentation[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2359-2367.
[8] BOLYA D，ZHOU C，XIAO F，et al. YOLACT：Real- time instance segmentation[C]// Proceedings of the IEEE International Conference on Computer Vision，2019：9157-9166.
[9] ARNOLD E，AL-JARRAH O Y，DIANATI M，et al. A survey on 3d object detection methods for autonomous driving applications[J]. IEEE Transactions on Intelligent Transportation Systems，2019，20(10)：3782-3795.
[10] FENG D，HAASE-SCHÜTZ C，ROSENBAUM L，et al. Deep multi-modal object detection and semantic segmentation for autonomous driving：Datasets，methods，and challenges[J]. IEEE Transactions on Intelligent Transportation Systems，2020，22(3)：1341-1360.
[11] STONE L D，STREIT R L，CORWIN T L，et al. Bayesian multiple target tracking[M]. Boston：Artech House，2013.
[12] 彭湃，耿可可，殷国栋，等. 基于传感器融合里程计的相机与激光雷达自动重标定方法[J]. 机械工程学报，2021，57(20)：206-214. PENG Pai，GENG Keke，YIN Guodong，et al. Automatic recalibration of camera and LiDAR using sensor fusion odometry[J]. Journal of Mechanical Engineering，2021，57(20)：206-214.
[13] FERGUSON D，DARMS M，URMSON C，et al. Detection，prediction，and avoidance of dynamic obstacles in urban environments[C]// IEEE Intelligent Vehicles Symposium，2008：1149-1154.
[14] BRADSKI G，KAEHLER A. Learning OpenCV：Computer vision with the OpenCV library[M]. Sebastopol：O’Reilly Media，2008.

[1]	FU Yang, ZHANG Yue, MAO Ying, TANG Xiaohua, CHEN Zugao, XU Hewu, YANG Yupei, GAO Bin, TIAN Guiyun. Feature Boosting Framework for Pipeline Multi-sensing Defects Inspection Using an Intelligent Pig System [J]. Journal of Mechanical Engineering, 2024, 60(20): 51-67.
[2]	GUAN Haijie, WANG Boyang, GONG Jianwei, CHEN Huiyan. Unified Motion Planning Method for Heterogeneous Vehicles [J]. Journal of Mechanical Engineering, 2024, 60(18): 288-298.
[3]	CHU Duanfeng, PENG Saiqian, HU Haiyang, PI Dawei. Predictive Driving Risk Field Model [J]. Journal of Mechanical Engineering, 2024, 60(10): 160-170.
[4]	LIN Chen, HE Zhicheng, HUANG Yifei, LIN Zhigui, FU Guang, HUANG Jin. Research on Detection Method for Driving Scenarios Based on Multi-stage Parameter Fusion Network [J]. Journal of Mechanical Engineering, 2024, 60(10): 64-75.
[5]	ZHANG Yu, TAN Zubing, CAO Dongpu, CHEN Long. Survey on Key Techniques for Visual and Inertial Based Odomety [J]. Journal of Mechanical Engineering, 2024, 60(10): 3-21.
[6]	ZENG Di, ZHENG Ling, LI Yinong, YANG Xiantong. Bayesian Inverse Reinforcement Learning-based Reward Learning for Automated Driving [J]. Journal of Mechanical Engineering, 2024, 60(10): 245-260.
[7]	CHEN Xiaoming, LI Bai, FAN Lili, WANG Yazhou, ZHANG Tantan, ZHANG Youmin, CAO Dongpu. High-performance Trajectory Optimization for Automated Parking via Half-space Constraining Theory [J]. Journal of Mechanical Engineering, 2024, 60(10): 273-288.
[8]	ZHAO Linfeng, FENG Xiao, FANG Ting, WANG Ning, CHEN Wuwei, WANG Huiran. Active Collision Avoidance Method of Intelligent Vehicles Based on the Trajectory Prediction of Front Vehicles in High-speed Scene [J]. Journal of Mechanical Engineering, 2024, 60(10): 289-301.
[9]	MAO Yangkun, DUAN Xianyin, LIN Xin, FUH Y H J, ZHU KunPeng. Melt Pool and Spatter Monitoring in Selective Laser Melting Forming Process Based on Target Detection [J]. Journal of Mechanical Engineering, 2023, 59(9): 335-348.
[10]	WANG Yong-sheng, LIU Jin-xin, BU De-xu, JIANG Fa-chao, LUO Yu-gong. Interactive Pedestrian Crossing Behavior Prediction under the Explicit Communication of Intelligent Vehicles [J]. Journal of Mechanical Engineering, 2023, 59(8): 151-162.
[11]	PENG Pai, GENG Keke, WANG Ziwei, LIU Zhichao, YIN Guodong. Review on Environmental Perception Methods of Autonomous Vehicles [J]. Journal of Mechanical Engineering, 2023, 59(20): 281-303.
[12]	ZANG Yong, CAI Yingfeng, SUN Xiaoqiang, XU Xing, CHEN Long, WANG Hai. Research on Intelligent Vehicle Trajectory Tracking Coordination Control Method Based on Extension Game [J]. Journal of Mechanical Engineering, 2022, 58(8): 181-194.
[13]	JIA Hanbing, LIU Peng, ZHANG Lei, WANG Zhenpo. Lane-changing Decision Model Development by Combining Rules Abstract and Machine Learning Technique [J]. Journal of Mechanical Engineering, 2022, 58(4): 212-221.
[14]	LIU Yonggang, YU Fengning, ZHANG Xinjie, CHEN Zheng, QIN Datong. Research on 3D Object Detection Based on Laser Point Cloud and Image Fusion [J]. Journal of Mechanical Engineering, 2022, 58(24): 289-299.
[15]	YANG Junru, CHU Duanfeng, LU Liping, WANG Jinxiang, WU Chaozhong, YIN Guodong. Review on Human-machine Shared Control of Intelligent Vehicles [J]. Journal of Mechanical Engineering, 2022, 58(18): 31-55.