Automatic Driving Decision Network of Crawler Excavator Based on Multi-modal

doi:10.3901/JME.2025.04.380

Abstract

Abstract: The application scenario of crawler excavators is complex. The working environment is irregular. Current deep learning-based autonomous driving decision-making methods mainly take monocular camera RGB images as input, have single data types, low prediction accuracy and insufficient understanding of driving scenes, which are not sufficient to complete autonomous driving decisions of electric construction machinery. In order to better realize the decision of automatic driving of electric construction machinery, multiple binocular image information is fused, and attention mechanism is employed to construct a multi-modal behavior decision model of electric construction machinery, and the multi-task predictors of steering and speed are obtained. Conduct testing using open-source driving scenario datasets and unstructured road real site datasets, and conduct real vehicle testing. The experimental results show that the model that integrates multiple binocular image information has obvious advantages in generalization ability in predicting the steering angle and speed of tracked excavators, and can effectively complete the automatic walking of tracked excavators and perform autonomous obstacle avoidance.

Key words: construction machinery, deep learning, autonomous driving, behavior decision, multi-modal

CLC Number:

TP242

CHEN Qihuai, LIN Tianliang, MA Ronghua, WEN Jianhe, MIAO Cheng, REN Haoling. Automatic Driving Decision Network of Crawler Excavator Based on Multi-modal[J]. Journal of Mechanical Engineering, 2025, 61(4): 380-388.

References

[1] PAREKH D，PODDARO N，RAJPURKAR A，et al. A review on autonomous vehicles:Progress，methods and challenges[J]. Electronics，2022，11(14):2162.
[2] LIU Q，LI X，YUAN S，et al. Decision-making technology for autonomous vehicles:Learning-based methods，applications and future outlook[C]//2021 IEEE International Intelligent Transportation Systems Conference (ITSC). Indianapolis，American:IEEE，2021:30-37.
[3] ERALIEV O，LEE K，SHIN D，et al. Sensing，perception，decision，planning and action of autonomous excavators[J]. Automation in Construction，2022，141: 104428.
[4] ZHANG L，ZHAO J，LONG P，et al. An autonomous excavator system for material loading tasks[J]. Science Robotics，2021，6(55):eabc3164.
[5] BOJARSKI M，DEL TESTA D，DWORAKOWSKI D，et al. End to end learning for self-driving cars[EB/OL]//[2016-04-25]. https://arxiv.org/pdf:1604.07316.
[6] CHEN Y，WANG J，LI J，et al. Lidar-video driving dataset:Learning driving policies effectively[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Salt Lake City American:IEEE，2018:5870-5878.
[7] FANG Z，LIN T，LI Z，et al. Automatic walking method of construction machinery based on binocular camera environment perception[J]. Micromachines，2022，13(5): 671.
[8] 林添良，姚瑜，许文杰，等. 基于环境识别的电动工程机械无人驾驶行走方法[J]. 机械工程学报，2021，57(10):1-8. LIN Tianliang，YAO Yu，XU Wenjie，et al. Driverless walking method of electric construction machinery based on environment recognition[J]. Journal of Mechanical Engineering，2021，57(10):1-8.
[9] YU C，WANG J，PENG C，et al. Bisenet:Bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV). Munich，German:Springer，2018:325-341.
[10] HE K，ZHANG X，REN S，et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas:IEEE，2016:770-778.
[11] VASWANI A，SHAZEER N，PARMAR N，et al. Attention is all you need[C]//Advances in Neural Information Processing Systems(NIPS). Long Beach: MIT Press，2017:5998-6008.
[12] HOU Q，ZHOU D，FENG J. Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR)，New York:IEEE，2021:13713-13722.
[13] IOFFE S，SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning(ICML). Lile:PMLR，2015:448-456.
[14] LOSHCHILOV I，HUTTER F. Fixing weight decay regularization in adam[EB/OL]. [2017-11-14]. https://arxiv.org/pdf/1711.05101.
[15] LOSHCHILOV I，HUTTER F. Sgdr: Stochastic gradient descent with warm restarts[EB/OL].[2016-08-13]. https://arxiv.org/pdf/1608.03983.
[16] 赵祥模，连心雨，刘占文，等. 基于MM-STConv的端到端自动驾驶行为决策模型[J].中国公路学报，2020，33(3):170-183. ZHAO Xiangmo，LIAN Xinyu，LIU Zhanwen，et al. End-to-end autonomous driving-behavior decision model based on MM-STConv[J]. Journal of China Highway Engineering，2020，33(3):170-183.