Multi-expert Learning Method for Autonomous Lane-changing Decision-making in Multi-scenario Highway Environments

doi:10.3901/JME.2025.22.198

Abstract

Abstract: The decision-making process of autonomous vehicles on highways involves a sequence of driving maneuvers aimed at improving safety and efficiency, which, however, results in considerable training time for the learning algorithm. This study proposes a multi-expert learning method（MELM） that integrates multiple actors（experts）, each trained using the soft actor-critic（SAC） algorithm under constraints derived from distinct sub-layer scenarios. Each sub-layer scenario is defined according to the distinct properties of the original training scenario. Each expert controls the vehicle in its corresponding sub-layer scenario and is integrated via a classifier that identifies the applicable sub-scenario. As a result, the MELM significantly reduces the model’s training time by 62.19% compared to a single SAC model, while also improving driving safety and efficiency, attributed to a remarkable reduction in the training difficulty of SAC. The proposed MELM is compared against several state-of-the-art methods under representative driving scenarios. Simulation results show a 27.06% improvement in driving efficiency compared to the single SAC model, along with high safety performance characterized by zero collision and off-road incidents across 100 testing episodes（～100 000 timesteps）. Furthermore, the adaptability of MELM is validated through simulation in a variety of scenarios with different condition settings.

Key words: autonomous driving, multi-experts learning method, lane changing decision-making

CLC Number:

TG156

YAO Fuxing, LI Haoyu, LENG Jianghao, YANG Xiongji, SUN Chao. Multi-expert Learning Method for Autonomous Lane-changing Decision-making in Multi-scenario Highway Environments[J]. Journal of Mechanical Engineering, 2025, 61(22): 198-210.

References

[1] HE Xiangkun, HUANG Wenhui, LÜChen. Toward trustworthy decision-making for autonomous vehicles:A robust reinforcement learning approach with safety guarantees[J]. Engineering,2024,33:77-89.
[2] LIU Jiaqi,QI Xiao,HANG Peng,et al. Enhancing social decision-making of autonomous vehicles:A mixedstrategy game approach with interaction orientation identification[J]. IEEE Transactions on Vehicular Technology,2024,73(6):7529-7539.
[3] HU Wen,WANG Cong,DENG Zejian,et al. Uncertaintyaware decision making and planning for ICV based on asymmetric driving aggressiveness[J]. IEEE Transactions on Intelligent Vehicles,2024,9(2):453-462.
[4] CHEN L,SHAN Y,TIAN W,et al. A fast and efficient double-tree RRT*-like sampling-based planner applying on mobile robotic systems[J]. IEEE/ASME Transactions on Mechatronics,2018,23(4):1643-1655.
[5] SCHWARTING W, ALONSO-MORA J, RUS D.Planning and decision-making for autonomous vehicles[J].Annual Review of Control,Robotics,and Autonomous Systems,2018,1(1):187-210.
[6] LI Tianyu,RUAN Junjie,ZHANG Kai. The investigation of reinforcement learning-based End-to-End decisionmaking algorithms for autonomous driving on the road with consecutive sharp turns[J]. Green Energy and Intelligent Transportation,2025,4(3):100288.
[7] NI Haoyu,YU Guojun,CHEN Pengcheng,et al. An integrated framework of lateral and longitudinal behavior decision-making for autonomous driving using reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2024,73(4):4213-4227.
[8] MOSLEM S,FAROOQ D,ESZTERGÁR-KISS D,et al.A novel spherical decision-making model for measuring the separateness of preferences for drivers'behavior factors associated with road traffic accidents[J]. Expert Systems with Applications,2024,238:122318.
[9] LI Liang,OTA K,DONG M. Humanlike driving:Empirical decision-making system for autonomous vehicles[J]. IEEE Transactions on Vehicular Technology,2018,67(8):6814-6823.
[10] KIRAN B R,SOBH I,TALPAERT V,et al. Deep reinforcement learning for autonomous driving:A survey[J]. IEEE Transactions on Intelligent Transportation Systems,2021,22(12):8492-8503.
[11] SAKIB N. Highway lane change under uncertainty with deep reinforcement learning based motion planner[J].IEEE Transactions on Intelligent Vehicles,2020,5(4):431-440.
[12] BI Xiaoyu,HE Mingyao,SUN Yuan,et al. Mix Qlearning for lane changing:A collaborative decisionmaking method in multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2025,74(2):2461-2472.
[13] FU Yuhang,LI Chenguang,YU F R,et al. A decisionmaking strategy for vehicle autonomous braking in emergency via deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology,2020,69(6):5876-5888.
[14] WANG Zhenyu,HUANG Haoxiang,TANG Jinyu,et al.A deep reinforcement learning-based approach for autonomous lane-changing velocity control in mixed flow of vehicle group level[J]. Expert Systems with Applications,2024,238:122158.
[15] WU Shuo, TIAN Dongqi, DUAN Xiangyu, et al.Continuous decision-making in lane changing and overtaking maneuvers for unmanned vehicles:A risk-aware reinforcement learning approach with task decomposition[J]. IEEE Transactions on Intelligent Vehicles,2024,9(1):183-196.
[16] HU Jie, ZHAO Wenlong, ZHENG Jiachen, et al.Interactive decision-making and trajectory planning for intelligent vehicles in lane change scenarios[J]. Journal of Mechanical Engineering,2025,61(10):322-334.
[17] WU Jun,YANG Hong,YANG Lin,et al. Human-guided deep reinforcement learning for optimal decision making of autonomous vehicles[J]. IEEE Transactions on Systems,Man,and Cybernetics:Systems,2024,54(9):4705-4716.
[18] WU Jun,HUANG Cheng,HUANG Haoxiang,et al.Recent advances in reinforcement learning-based autonomous driving behavior planning:A survey[J].Transportation Research Part C:Emerging Technologies,2024,164:104654.
[19] YUAN Kang,HUANG Yanjun,YANG Shuo,et al.Evolutionary decision-making and planning for autonomous driving based on safe and rational exploration and exploitation[J]. Engineering,2024,33:108-120.
[20] LIU Haochen, HUANG Zhiyu, MO Xiaoyu, et al.Augmenting reinforcement learning with transformerbased scene representation learning for decision-making of autonomous driving[J]. IEEE Transactions on Intelligent Vehicles,2024,9(3):575-587.
[21] YANG Chenguang, YUAN Kun,ZHU Qiang et al.Multi-expert learning of adaptive legged locomotion[J].Science Robotics,2020,5(49):2174.
[22] NGUYEN T T,NGUYEN N D,NAHAVANDI S. Deep reinforcement learning for multiagent systems:A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics,2020,50(9):3826-3839.
[23] TANG Xueyuan,CHEN Jing,PU Hu,et al. Double deep reinforcement learning-based energy management for a parallel hybrid electric vehicle with engine start-stop strategy[J]. IEEE Transactions on Transportation Electrification,2021,8(1):1376-1388.
[24] CHEN Xin,YANG Jie,ZHAI Chen,et al. Economic adaptive cruise control for electric vehicles based on ADHDP in a car-following scenario[J]. IEEE Access,2021,9:74949-74958.
[25] TANG Xueyuan, JIA Tianze, HU Xiaolei, et al.Naturalistic data-driven predictive energy management for plug-in hybrid electric vehicles[J]. IEEE Transactions on Transportation Electrification,2020,7(2):497-508.
[26] SHI Tianqi,WANG Peng,CHENG Xinyu,et al. Driving decision and control for automated lane change behavior based on deep reinforcement learning[C]//IEEE Intelligent Transportation Systems Conference (ITSC), 2019:2895-2900.
[27] YE Feng,CHENG Xinyu,WANG Peng,et al. Automated lane change strategy using proximal policy optimizationbased deep reinforcement learning[C]//IEEE Intelligent Vehicles Symposium (IV),2020:1746-1752.
[28] WANG Peng,LI Haoxiang,CHAN C Y. Continuous control for automated lane change behavior based on deep deterministic policy gradient algorithm[C]//IEEE Intelligent Vehicles Symposium (IV),2019:1454-1460.
[29] WANG Peng,LI Haoxiang,CHAN C Y. Quadratic Q-network for learning continuous control for autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems,2019,20(7):2445-2455.
[30] LI Xia,QIU Xian,WANG Jun,et al. A deep reinforcement learning-based approach for autonomous overtaking[C]//IEEE International Conference on Communications Workshops (ICC Workshops),2020:1-5.
[31] ZHANG Sibo, ZHUANG Weijian, LI Bo, et al.Integration of planning and deep reinforcement learning in speed and lane change decision-making for highway autonomous driving[J]. IEEE Transactions on Transportation Electrification,2024,10(2):1567-1579.
[32] 王慧然,王其东,陈无畏,等.基于车辆行驶安全边界的换道控制[J].机械工程学报,2020,56(18):143-153.WANG Huiran,WANG Qidong,CHEN Wuwei,et al.Lane change control based on vehicle driving safety boundaries[J]. Journal of Mechanical Engineering,2020,56(18):143-153.
[33] HU Wen,DENG Zejian,ZHANG Bangji,et al. Lane change decision-making and planning method for autonomous heavy trucks in dense traffic scenarios[J].Journal of Mechanical Engineering,2023,59(12):332-342.
[34] KONG J, PFEIFFER M, SCHILDBACH G, et al.Kinematic and dynamic vehicle models for autonomous driving control design[C]//IEEE Intelligent Vehicles Symposium (IV),2015:1094-1099.
[35] ATAEI M,KHAJEPOUR A,JEON S. Reconfigurable integrated stability control for four-and three-wheeled urban vehicles with flexible combinations of actuation systems[J]. IEEE/ASME Transactions on Mechatronics,2018,23(5):2031-2041.
[36] TREIBER M,HENNECKE A,HELBING D. Congested traffic states in empirical observations and microscopic simulations[J]. Physical Review E, 2000, 62(2):1805-1814.
[37] KESTING A, TREIBER M, HELBING D. General lane-changing model MOBIL for car-following models[J].Transportation Research Record,2007,1999(1):86-94.
[38] ZHOU Meng,QU Xiaobo,JIN Shuang. On the impact of cooperative autonomous vehicles in improving freeway merging:A modified intelligent driver model-based approach[J]. IEEE Transactions on Intelligent Transportation Systems,2016,18(6):1422-1428.