基于运动风险的强化学习换道决策方法研究

doi:10.3901/JME.2025.18.252

摘要/Abstract

摘要： 针对目前换道决策模型在稳定性、控制可靠性和场景适应性上的不足，提出一种基于运动风险的强化学习换道决策方法。首先基于换道最小安全距离理论建立运动风险模型，以有效整合驾驶场景信息，提高模型训练效率及稳定性。基于多场景的强化学习换道决策训练模型，以风险模型作为智能体的观测状态，并设计回报函数驱使智能体生成安全换道决策，然后通过仿真测试，将所设计的训练模型与普通的决策模型以及传统的基于速度和距离的决策模型的对比分析，验证所提算法在模型收敛速度、平均回报及成功率都有着更好的表现。最后通过仿真软件搭建结构化道路下典型的换道场景并进行仿真实验，结果表明，该算法能在各车速条件下规划出安全平滑的换道轨迹，同时满足舒适性和目标车速要求。

关键词: 汽车工程, 强化学习, 换道决策, DDQN, 运动风险

Abstract: According to the shortcomings of current lane change decision models in stability, control reliability and scene adaptability, a reinforcement learning lane change decision method based on motion risk is proposed to solve the problem. Firstly, a motion risk model is established based on the minimum safe distance theory of lane change to integrate driving scene information effectively and improve the training efficiency and stability of the model. The risk model is used as the observed state of the agent which is based on multi-scenario reinforcement learning training model of lane change decision, and the reward function is designed to drive the agent to generate safe lane change decision making. Then, through the simulation tests, the designed training model is compared with the ordinary decision making model and the traditional decision making model based on speed and distance, and the result shows that the proposed algorithm has much better performance in model convergence speed, average returns, and success rates. Finally, simulation experiments are conducted on typical lane change scenarios under structured roads based on simulation software. The result shows that the proposed algorithm can not only plan safe and smooth lane change trajectories under various speed conditions,but also meet the requirements of comfort and target speed at the same time.

Key words: automotive engineering, reinforcement learning, lane change decision, DDQN, sports risk

中图分类号:

U467

林歆悠, 代军, 曾松榕. 基于运动风险的强化学习换道决策方法研究[J]. 机械工程学报, 2025, 61(18): 252-266.

LIN Xinyou, DAI Jun, ZENG Songrong. Research on Lane Change Decision Method of Reinforcement Learning Based on Motion Risk[J]. Journal of Mechanical Engineering, 2025, 61(18): 252-266.

参考文献

[1] ZHANG Junyou，LIAO Yaping，WANG Shufeng，et al. Study on driving decision-making mechanism of autonomous vehicle based on an optimized support vector machine regression[J]. Applied Sciences，2018，8(1)：13.
[2] LIU Yonggang，WANG Xiao，LI Liang，et al. A novel lane change decision-making model of autonomous vehicle based on support vector machine[J]. IEEE Access，2019，7：26543-26550.
[3] 谷新平，韩云鹏，于俊甫. 基于决策机理与支持向量机的车辆换道决策模型[J]. 哈尔滨工业大学学报，2020，52(7)：111-121. GU Xinping，HAN Yunpeng，YU Junfu. Vehicle lane change decision model based on decision mechanism and support vector machine[J]. Journal of Harbin Institute of Technology，2020，52(27)：111-121.
[4] 贾寒冰，刘鹏，张雷，等. 基于规则与机器学习融合的换道决策建模方法研究[J]. 机械工程学报，2022，58(4)：212-221. JIA Hanbing，LIU Peng，ZHUANG Lei，et al. Lane- changing decision model development by combining rules abstract and machine learning technique[J]. Journal of Mechanical Engineering，2022，58(4)：212-221.
[5] Hou Y，Edara P，Sun C. Modeling mandatory lane changing using bayes classifier and decision trees[J]. IEEE Transactions on Intelligent Transportation Systems，2014，15(2)：647-655.
[6] Hou Y，Edara P，Sun C. Situation assessment and decision making for lane change assistance using ensemble learning methods[J]. Expert Systems with Applications，2015，8(42)：3875-3882.
[7] LIU Meiyu，SHI Jing. A cellular automata traffic flow model combined with a BP neural network based microscopic lane changing decision model[J]. Journal of Intelligent Transportation Systems，2019，23(4)：309-318.
[8] PENG Jinshuan，GUO Yingshi，FU Rui，et al. Multi-parameter prediction of drivers' lane-changing behaviour with neural network model[J]. Applied Ergonomics，2015，50：207-217.
[9] 房哲哲. 基于深度学习的换道行为建模与分析[D]. 北京：北京交通大学，2018. FANG Zhezhe. Modeling and analysis of lane-changing behavior based on deep learning[D]. Beijing：Beijing Jiaotong University，2018.
[10] 张羽翔，何钢磊，李鑫，等. 基于参数描述的换道场景自动驾驶精确决策学习[J]. 同济大学学报(自然科学版)，2021，49(增刊1)：132-140. ZHANG Yuxiang，HE Ganglei，LI Xin，et al. Accurate decision-making learning of autopilot in lane-changing scene based on parameter description[J]. Journal of Tongji University (Natural Science Edition)，2021，49(Suppl.1)：132-140.
[11] 赵树恩，王金祥，李玉玲. 基于多目标优化的智能车辆换道轨迹规划[J]. 交通运输工程学报，2021，21(2)：232-242. ZHAO Shuen，WANG Jinxiang，LI Yuling. Lane changing trajectory planning of intelligent vehicle based on multiple objective optimization[J]. Journal of Traffic and Transportation Engineering，2021，21(2)：232-242.
[12] 高凯，李勋豪，胡林，等. 基于多头注意力的CNN-LSTM的换道意图预测[J]. 机械工程学报，2022，58(22)：369-378. GAO Kai，LI Xunhao，HU Lin，et al. Lane change intention prediction of CNN-LSTM based on multi-head attention[J]. Journal of Mechanical Engineering，2022，58(22)：369-378.
[13] 吕超，鲁洪良，于洋，等. 基于分层强化学习和社会偏好的自主超车决策系统[J]. 中国公路学报，2022，35(3)：115-126. Lü Chao，LU Hongliang，YU Yang，et al. Autonomous overtaking decision making system based on hierarchical reinforcement learning and social preferences[J]. China Journal of Highway and Transport，2022，35(3)：115-126.
[14] GU Xinping，HAN Yunpeng，YU Junfu. A novel lane-changing decision model for autonomous vehicles based on deep autoencoder network and XGBoost[J]. IEEE Access，2020(8)：9846-9863.
[15] 裴晓飞，莫烁杰，陈祯福，等. 基于TD3算法的人机混驾交通环境自动驾驶汽车换道研究[J]. 中国公路学报，2021，34(11)：246-254. PEI Xiaofei，MO Shuojie，CHEN Zhenfu，et al. Lane changing of autonomous vehicle based on TD3 algorithm in human-machine hybrid driving environment[J]. China Journal of Highway and Transport，2021，34(11)：246-254.
[16] 罗鹏. 基于深度强化学习的智能车驾驶行为决策研究[D]. 武汉：武汉理工大学，2021. LUO Peng. Research on Intelligent vehicle driving behavior decision-making based on deep reinforcement learning[D]. Wuhan：Wuhan University of Technology，2021.
[17] Liao Jiangdong，Liu Teng，Tang Xiaolin，et al. Decision-making strategy on highway for autonomous vehicles using deep reinforcement learning[J]. IEEE Access，2020(8)：177804-177814.
[18] 唐双. 基于深度强化学习的智能车高速公路合流区换道决策研究[D]. 重庆：重庆大学，2020. TANG Shuang. Research on lane change decision of intelligent vehicle expressway confluence area based on deep reinforcement learning[D]. Chongqing：Chongqing University，2020.
[19] LI Guofa，YANG Yifan，LI Shen，et al. Decision making of autonomous vehicles in lane change scenarios：Deep reinforcement learning approaches with risk Awareness[J]. Transportation Research Part C-emerging Technologies，2022，134：103452.
[20] LV Kexuan，PEI Xiaofei，CHEN Ci，et al. A safe and efficient lane change decision-making strategy of autonomous driving based on deep reinforcement learning[J]. Mathematics，2022，10(9)：1551.
[21] 吕维. 基于深度增强学习的智能车安全并道决策研究[D]. 成都：电子科技大学，2019. Lü Wei. Research on safety parallel decision-making of Intelligent vehicle based on deep reinforcement learning[D]. Chengdu：University of Electronic Science and Technology of China，2019.
[22] Werling M，Ziegler J，Kammel S，et al. Optimal trajectory generation for dynamic street scenarios in a frenet frame[J]. 2010 IEEE International Conference on Robotics and Automation (ICRA)，2010：987-993.
[23] 黄晶，蓟仲勋，彭晓燕，等. 考虑驾驶人风格的换道轨迹规划与控制[J]. 中国公路学报，2019，32(6)：226-239. HUANG Jing，JI Zhongxun，PENG Xiaoyan，et al. Consider the driver’s style of lane change trajectory planning and control[J]. Journal of Highway，China，2019，32(6)：226-239.
[24] LARNER D L R J S. Method and system for determining and dynamically updating a route and driving style for passenger comfot：US11585670[P]. 2018-10-23.
[25] 张荣辉，游峰，初鑫男，等. 车-车协同下无人驾驶车辆的换道汇入控制方法[J]. 中国公路学报，2018，31(4)：180-191. ZHANG Ronghui，YOU Feng，CHU Xinnan，et al. Control method of lane exchange of driverless vehicles under vehicle-vehicle coordination[J]. Journal of Highway，China，2018，31(4)：180-191.