面向多场景高速公路的多专家学习自主换道决策方法

doi:10.3901/JME.2025.22.198

摘要/Abstract

摘要： 高速公路环境下自动驾驶汽车的自主换道决策过程对驾驶安全性与效率有很高要求，导致基于学习的决策算法训练耗时显著增加。研究提出一种多专家学习换道决策方法(Multi-experts learning method,MELM)，该方法集成多个软演员评论家算法(Soft actor-critic,SAC)训练的专家模型，每个专家在预先划分的不同子场景约束条件下进行训练。子场景根据原始高速公路的换道特征定义，各专家在其对应子场景中控制车辆，并通过场景分类器识别当前子场景进行集成。相较于单一SAC模型，MELM能够显著降低SAC训练难度，使模型训练时间减少62.19%。仿真实验结果表明：在典型驾驶场景下MELM相比单一SAC模型驾驶效率提高27.06%，且在100回合测试(约10万步)中达成零碰撞与零出界事故的优秀安全性能。此外，通过不同道路条件设置下的仿真试验场景验证了MELM的高适应性。

关键词: 自动驾驶, 多专家学习方法, 自主换道决策

Abstract: The decision-making process of autonomous vehicles on highways involves a sequence of driving maneuvers aimed at improving safety and efficiency, which, however, results in considerable training time for the learning algorithm. This study proposes a multi-expert learning method（MELM） that integrates multiple actors（experts）, each trained using the soft actor-critic（SAC） algorithm under constraints derived from distinct sub-layer scenarios. Each sub-layer scenario is defined according to the distinct properties of the original training scenario. Each expert controls the vehicle in its corresponding sub-layer scenario and is integrated via a classifier that identifies the applicable sub-scenario. As a result, the MELM significantly reduces the model’s training time by 62.19% compared to a single SAC model, while also improving driving safety and efficiency, attributed to a remarkable reduction in the training difficulty of SAC. The proposed MELM is compared against several state-of-the-art methods under representative driving scenarios. Simulation results show a 27.06% improvement in driving efficiency compared to the single SAC model, along with high safety performance characterized by zero collision and off-road incidents across 100 testing episodes（～100 000 timesteps）. Furthermore, the adaptability of MELM is validated through simulation in a variety of scenarios with different condition settings.

Key words: autonomous driving, multi-experts learning method, lane changing decision-making

中图分类号:

TG156

姚福星, 李浩宇, 冷江昊, 杨雄基, 孙超. 面向多场景高速公路的多专家学习自主换道决策方法[J]. 机械工程学报, 2025, 61(22): 198-210.

YAO Fuxing, LI Haoyu, LENG Jianghao, YANG Xiongji, SUN Chao. Multi-expert Learning Method for Autonomous Lane-changing Decision-making in Multi-scenario Highway Environments[J]. Journal of Mechanical Engineering, 2025, 61(22): 198-210.

参考文献

[1] HE Xiangkun, HUANG Wenhui, LÜChen. Toward trustworthy decision-making for autonomous vehicles:A robust reinforcement learning approach with safety guarantees[J]. Engineering,2024,33:77-89.
[2] LIU Jiaqi,QI Xiao,HANG Peng,et al. Enhancing social decision-making of autonomous vehicles:A mixedstrategy game approach with interaction orientation identification[J]. IEEE Transactions on Vehicular Technology,2024,73(6):7529-7539.
[3] HU Wen,WANG Cong,DENG Zejian,et al. Uncertaintyaware decision making and planning for ICV based on asymmetric driving aggressiveness[J]. IEEE Transactions on Intelligent Vehicles,2024,9(2):453-462.
[4] CHEN L,SHAN Y,TIAN W,et al. A fast and efficient double-tree RRT*-like sampling-based planner applying on mobile robotic systems[J]. IEEE/ASME Transactions on Mechatronics,2018,23(4):1643-1655.
[5] SCHWARTING W, ALONSO-MORA J, RUS D.Planning and decision-making for autonomous vehicles[J].Annual Review of Control,Robotics,and Autonomous Systems,2018,1(1):187-210.
[6] LI Tianyu,RUAN Junjie,ZHANG Kai. The investigation of reinforcement learning-based End-to-End decisionmaking algorithms for autonomous driving on the road with consecutive sharp turns[J]. Green Energy and Intelligent Transportation,2025,4(3):100288.
[7] NI Haoyu,YU Guojun,CHEN Pengcheng,et al. An integrated framework of lateral and longitudinal behavior decision-making for autonomous driving using reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2024,73(4):4213-4227.
[8] MOSLEM S,FAROOQ D,ESZTERGÁR-KISS D,et al.A novel spherical decision-making model for measuring the separateness of preferences for drivers'behavior factors associated with road traffic accidents[J]. Expert Systems with Applications,2024,238:122318.
[9] LI Liang,OTA K,DONG M. Humanlike driving:Empirical decision-making system for autonomous vehicles[J]. IEEE Transactions on Vehicular Technology,2018,67(8):6814-6823.
[10] KIRAN B R,SOBH I,TALPAERT V,et al. Deep reinforcement learning for autonomous driving:A survey[J]. IEEE Transactions on Intelligent Transportation Systems,2021,22(12):8492-8503.
[11] SAKIB N. Highway lane change under uncertainty with deep reinforcement learning based motion planner[J].IEEE Transactions on Intelligent Vehicles,2020,5(4):431-440.
[12] BI Xiaoyu,HE Mingyao,SUN Yuan,et al. Mix Qlearning for lane changing:A collaborative decisionmaking method in multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2025,74(2):2461-2472.
[13] FU Yuhang,LI Chenguang,YU F R,et al. A decisionmaking strategy for vehicle autonomous braking in emergency via deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2020,69(6):5876-5888.
[14] WANG Zhenyu,HUANG Haoxiang,TANG Jinyu,et al.A deep reinforcement learning-based approach for autonomous lane-changing velocity control in mixed flow of vehicle group level[J]. Expert Systems with Applications,2024,238:122158.
[15] WU Shuo, TIAN Dongqi, DUAN Xiangyu, et al.Continuous decision-making in lane changing and overtaking maneuvers for unmanned vehicles:A risk-aware reinforcement learning approach with task decomposition[J]. IEEE Transactions on Intelligent Vehicles,2024,9(1):183-196.
[16] HU Jie, ZHAO Wenlong, ZHENG Jiachen, et al.Interactive decision-making and trajectory planning for intelligent vehicles in lane change scenarios[J]. Journal of Mechanical Engineering,2025,61(10):322-334.
[17] WU Jun,YANG Hong,YANG Lin,et al. Human-guided deep reinforcement learning for optimal decision making of autonomous vehicles[J]. IEEE Transactions on Systems,Man,and Cybernetics:Systems,2024,54(9):4705-4716.
[18] WU Jun,HUANG Cheng,HUANG Haoxiang,et al.Recent advances in reinforcement learning-based autonomous driving behavior planning:A survey[J].Transportation Research Part C:Emerging Technologies,2024,164:104654.
[19] YUAN Kang,HUANG Yanjun,YANG Shuo,et al.Evolutionary decision-making and planning for autonomous driving based on safe and rational exploration and exploitation[J]. Engineering,2024,33:108-120.
[20] LIU Haochen, HUANG Zhiyu, MO Xiaoyu, et al.Augmenting reinforcement learning with transformerbased scene representation learning for decision-making of autonomous driving[J]. IEEE Transactions on Intelligent Vehicles,2024,9(3):575-587.
[21] YANG Chenguang, YUAN Kun,ZHU Qiang et al.Multi-expert learning of adaptive legged locomotion[J].Science Robotics,2020,5(49):2174.
[22] NGUYEN T T,NGUYEN N D,NAHAVANDI S. Deep reinforcement learning for multiagent systems:A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics,2020,50(9):3826-3839.
[23] TANG Xueyuan,CHEN Jing,PU Hu,et al. Double deep reinforcement learning-based energy management for a parallel hybrid electric vehicle with engine start-stop strategy[J]. IEEE Transactions on Transportation Electrification,2021,8(1):1376-1388.
[24] CHEN Xin,YANG Jie,ZHAI Chen,et al. Economic adaptive cruise control for electric vehicles based on ADHDP in a car-following scenario[J]. IEEE Access,2021,9:74949-74958.
[25] TANG Xueyuan, JIA Tianze, HU Xiaolei, et al.Naturalistic data-driven predictive energy management for plug-in hybrid electric vehicles[J]. IEEE Transactions on Transportation Electrification,2020,7(2):497-508.
[26] SHI Tianqi,WANG Peng,CHENG Xinyu,et al. Driving decision and control for automated lane change behavior based on deep reinforcement learning[C]//IEEE Intelligent Transportation Systems Conference (ITSC), 2019:2895-2900.
[27] YE Feng,CHENG Xinyu,WANG Peng,et al. Automated lane change strategy using proximal policy optimizationbased deep reinforcement learning[C]//IEEE Intelligent Vehicles Symposium (IV),2020:1746-1752.
[28] WANG Peng,LI Haoxiang,CHAN C Y. Continuous control for automated lane change behavior based on deep deterministic policy gradient algorithm[C]//IEEE Intelligent Vehicles Symposium (IV),2019:1454-1460.
[29] WANG Peng,LI Haoxiang,CHAN C Y. Quadratic Q-network for learning continuous control for autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems,2019,20(7):2445-2455.
[30] LI Xia,QIU Xian,WANG Jun,et al. A deep reinforcement learning-based approach for autonomous overtaking[C]//IEEE International Conference on Communications Workshops (ICC Workshops),2020:1-5.
[31] ZHANG Sibo, ZHUANG Weijian, LI Bo, et al.Integration of planning and deep reinforcement learning in speed and lane change decision-making for highway autonomous driving[J]. IEEE Transactions on Transportation Electrification,2024,10(2):1567-1579.
[32] 王慧然,王其东,陈无畏,等.基于车辆行驶安全边界的换道控制[J].机械工程学报,2020,56(18):143-153.WANG Huiran,WANG Qidong,CHEN Wuwei,et al.Lane change control based on vehicle driving safety boundaries[J]. Journal of Mechanical Engineering,2020,56(18):143-153.
[33] HU Wen,DENG Zejian,ZHANG Bangji,et al. Lane change decision-making and planning method for autonomous heavy trucks in dense traffic scenarios[J].Journal of Mechanical Engineering,2023,59(12):332-342.
[34] KONG J, PFEIFFER M, SCHILDBACH G, et al.Kinematic and dynamic vehicle models for autonomous driving control design[C]//IEEE Intelligent Vehicles Symposium (IV),2015:1094-1099.
[35] ATAEI M,KHAJEPOUR A,JEON S. Reconfigurable integrated stability control for four-and three-wheeled urban vehicles with flexible combinations of actuation systems[J]. IEEE/ASME Transactions on Mechatronics,2018,23(5):2031-2041.
[36] TREIBER M,HENNECKE A,HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Physical Review E, 2000, 62(2):1805-1814.
[37] KESTING A, TREIBER M, HELBING D. General lane-changing model MOBIL for car-following models[J].Transportation Research Record,2007,1999(1):86-94.
[38] ZHOU Meng,QU Xiaobo,JIN Shuang. On the impact of cooperative autonomous vehicles in improving freeway merging:A modified intelligent driver model-based approach[J]. IEEE Transactions on Intelligent Transportation Systems,2016,18(6):1422-1428.