基于时序差分学习模型预测控制的一体化自动驾驶换道策略

doi:10.3901/JME.2024.10.329

机械工程学报 ›› 2024, Vol. 60 ›› Issue (10): 329-338.doi: 10.3901/JME.2024.10.329

扫码分享

基于时序差分学习模型预测控制的一体化自动驾驶换道策略

杨硕¹, 李时珍¹, 赵中原², 黄小鹏³, 黄岩军¹

1. 同济大学汽车学院上海 201804;
2. 南京信息工程大学自动化学院南京 210044;
3. 中电科东方通信集团有限公司广东 519060

收稿日期:2023-08-15 修回日期:2023-12-26 出版日期:2024-05-20 发布日期:2024-07-24
作者简介:杨硕,男,1995年出生,博士研究生。主要研究方向为自动驾驶汽车,强化学习,智能交通系统和车辆动力学。
E-mail:yangshuo_jlu@163.com
黄岩军(通信作者),男,1986年出生,博士,教授,博士研究生导师。主要研究方向为自动驾驶和人工智能相结合的决策规划、运动控制、人机合作驾驶。
E-mail:yanjun_huang@tongji.edu.cn
基金资助:
国家自然科学基金-企业创新发展联合基金资助项目(U23B2061)。

Integrated Autonomous Driving Lane Change Policy Based on Temporal Difference Learning Model Predictive Control

YANG Shuo¹, LI Shizhen¹, ZHAO Zhongyuan², HUANG Xiaopeng³, HUANG Yanjun¹

1. School of Automotive Studies, Tongji University, Shanghai 201804;
2. College of Automation, Nanjing University of Information Science and Technology, Nanjing 210044;
3. China Electronics Technology Eastern Communication Group Co., Ltd., Guangdong 519060

Received:2023-08-15 Revised:2023-12-26 Online:2024-05-20 Published:2024-07-24

摘要/Abstract

摘要： 具有自进化能力的自动驾驶换道策略有望在复杂开放的交通环境中提升性能，以应对更多的未知场景。时序差分学习模型预测控制(Temporal difference learning for model predictive control, TD-MPC)结合有模型和无模型强化学习方法的优势，具有学习效率高、性能优异的特点。基于此，为了提高自动驾驶换道策略的整体性能，提出基于TD-MPC的自动驾驶一体化换道策略。具体来说，针对自动换道问题，提出基于驾驶倾向网络的一体化自动驾驶换道策略架构，构建强化学习问题并设计完备的奖励函数，对决策规划优化问题进行统一求解。应用TD-MPC算法设计内部模型来预测未来状态和奖励，实现短时域内的局部轨迹优化，同时使用时序差分学习实现对长期汇报的估计，以得到驾驶倾向网络参数。所提出方法在高保真仿真环境中被验证，结果表明，所提出方法相比规则方案保证行驶效率，并且提高安全性和舒适性。同时与软演员-评论家算法(Soft actor critic, SAC)相比，实现了7～9倍的学习效率提升。

关键词: 自动驾驶, 强化学习, 一体化决策规划

Abstract: Autonomous vehicles are expected to achieve self-evolution in the real-world environment to gradually cover more complex and changing scenarios. Temporal difference learning for model predictive control(TD-MPC) combines the advantages of model-free and model-free reinforcement learning methods, and has the characteristics of high learning efficiency and excellent performance. Based on this, in order to improve the overall performance of the automated lane change policy, an integrated automated lane change method based on TD-MPC is proposed. Specifically, an integrated architecture based on driving propensity network is proposed. The reinforcement learning problem is constructed and a complete reward function is designed to solve the decision planning optimization problem in a unified way. The TD-MPC algorithm is used to design an internal model to predict the future state and reward, so as to realize the local trajectory optimization in the short time domain. At the same time, the temporal difference learning is used to estimate the long-term report to obtain the parameters of the driving tendency network. The proposed method is verified in a high-fidelity simulation environment. The results show that compared with the regular scheme, the proposed method ensures driving efficiency, and improves safety and comfort. At the same time, compared with the soft actor critic(SAC) algorithm, the learning efficiency is improved by 7 to 9 times.

Key words: autonomous driving, reinforcement learning, integrated decision making and planning

中图分类号:

U461

杨硕, 李时珍, 赵中原, 黄小鹏, 黄岩军. 基于时序差分学习模型预测控制的一体化自动驾驶换道策略[J]. 机械工程学报, 2024, 60(10): 329-338.

YANG Shuo, LI Shizhen, ZHAO Zhongyuan, HUANG Xiaopeng, HUANG Yanjun. Integrated Autonomous Driving Lane Change Policy Based on Temporal Difference Learning Model Predictive Control[J]. Journal of Mechanical Engineering, 2024, 60(10): 329-338.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://www.cjmenet.com.cn/CN/10.3901/JME.2024.10.329

http://www.cjmenet.com.cn/CN/Y2024/V60/I10/329

参考文献

[1] 林歆悠，叶卓明，周斌豪. 基于DQN强化学习的自动驾驶转向控制策略[J]. 机械工程学报，2023，59(16):315-324. LIN Xinyou，YE Zhuoming，ZHOU Binhao. DQN reinforcement learning-based steering control strategy for autonomous driving[J]. Journal of Mechanical Engineering，2023，59(16):315-324.
[2] WANG Liwen，YANG Shuo，YUAN Kang，et al. A Combined reinforcement learning and model predictive control for car-following maneuver of autonomous vehicles[J]. Chinese Journal of Mechanical Engineering，2023，36(1):80.
[3] JASWANTH M，NARAYANA N K L，RAHUL S，et al. Autonomous car controller using behaviour planning based on finite state machine[C]//2022 6th International Conference on Trends in Electronics and Informatics (ICOEI). IEEE，2022:296-302.
[4] WANG Xuanyu，QI Xudong，WANG Ping，et al. Decision making framework for autonomous vehicles driving behavior in complex scenarios via hierarchical state machine[J]. Autonomous Intelligent Systems，2021，1:1-12.
[5] ZHANG Chaoyong，CHU Duanfeng，LIU Shidong，et al. Trajectory planning and tracking for autonomous vehicle based on state lattice and model predictive control[J]. IEEE Intelligent Transportation systems magazine，2019，11(2):29-40.
[6] MCNAUGHTON M，URMSON C，DOLAN J M，et al. Motion planning for autonomous driving with a conformal spatiotemporal lattice[C]//2011 IEEE International Conference on Robotics and Automation. IEEE，2011:4889-4895.
[7] KATRAKAZAS C，QUDDUS M，CHEN W H，et al. Real-time motion planning methods for autonomous on-road driving:State-of-the-art and future research directions[J]. Transportation Research Part C:Emerging Technologies，2015，60:416-442.
[8] JI Jie，KHAJEPOUR A，MELEK W W，et al. Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints[J]. IEEE Transactions on Vehicular Technology，2016，66(2):952-964.
[9] LI Bai，ZHANG Youmin，FENG Yiheng，et al. Balancing computation speed and quality:A decentralized motion planning method for cooperative lane changes of connected and automated vehicles[J]. IEEE Transactions on Intelligent Vehicles，2018，3(3):340-350.
[10] 杜荣华，胡鸿飞，高凯，等. 基于变预测时域MPC的自动驾驶汽车轨迹跟踪控制研究[J]. 机械工程学报，2022，58(24):275-288. DU Ronghua，HU Hongfei，GAO Kai，etal. Research on trajectory tracking control of autonomous vehicle based on MPC with variable predictive horizon[J]. Journal of Mechanical Engineering，2022，58(24):275-288.
[11] 高振海，闫相同，高菲. 基于逆向强化学习的纵向自动驾驶决策方法[J]. 汽车工程，2022，44(7):969-975. GAO Zhenhai，YAN Xiangtong，GAO Fei. A decision-making method for longitudinal autonomous driving based on inverse reinforcement learning[J]. Automotive Engineering，2022，44(7):969-975.
[12] SEWAK M. Deep reinforcement learning[M]. Singapore:Springer Singapore，2019.
[13] AL-SHARMAN M，DEMPSTER R，DAOUD M A，et al. Self-learned autonomous driving at unsignalized intersections:A hierarchical reinforced learning approach for feasible decision-making[J]. IEEE Transactions on Intelligent Transportation Systems，2023，24(11):12345-1235.
[14] LI Guofa，YANG Yifan，LI Shen，et al. Decision making of autonomous vehicles in lane change scenarios:Deep reinforcement learning approaches with risk awareness[J]. Transportation research part C:Emerging Technologies，2022，134:103452.
[15] 张志勇，黄大洋，黄彩霞，等. TD3算法改进与自动驾驶汽车并道策略学习[J]. 机械工程学报，2023，59(8):224-234. ZHANG Zhiyong，HUANG Dayang，HUANG Caixia，et al. TD3 algorithm improving and lane-merging strategy learning for autonomous vehicles[J]. Journal of Mechanical Engineering，2023，59(8):224-234.
[16] 杨惟轶，白辰甲，蔡超，等. 深度强化学习中稀疏奖励问题研究综述[J]. 计算机科学，2020，47(3):182-191. YANG Weiyi，BAI Chenjia，CAI Chao，et al. Survey on sparse reward in deep reinforcement learning[J]. Computer Science，2020，47(3):182-191.
[17] HUBMANN C，BECKER M，ALTHOFF D，et al. Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles[C]//2017 IEEE Intelligent Vehicles Symposium (IV). IEEE，2017:1671-1678.
[18] 张珂，刘畅，兰鹏宇. 基于改进人工势场法的局部路径规划[J]. 汽车文摘，2021(7):59-62. ZHANG Ke，LIU Chang，LAN Pengyu. Local path planning based on improved artificial potential field method [J]. Automotive Digest，2021(7):59-62.
[19] HANSEN N，WANG Xiaolong，SU Hao. Temporal difference learning for model predictive control[J/OL]. arXiv Preprint arXiv:2203.04955，[2022-03-09]. https://arxiv.org/abs/2203.04955.
[20] WILLIAMS G，ALDRICH A，THEODOROU E. Model predictive path integral control using covariance variable importance sampling[J]. arXiv Preprint arXiv:1509.01149，2015.
[21] MOGHADAM M，ALIZADEH A，TEKIN E，et al. An end-to-end deep reinforcement learning approach for the long-term short-term planning on the frenet space[J].
arXiv Preprint arXiv:2011.13098，2020.
[22] DOSOVITSKIY A，ROS G，CODEVILLA F，et al. CARLA:An open urban driving simulator[C]//Conference on Robot Learning. PMLR，2017:1-16.
[23] HAARNOJA T，ZHOU A，HARTIKAINEN K，et al. Soft actor-critic algorithms and applications[J/OL]. arXiv Preprint arXiv:1812.05905，[2018-12-13]. https://arxiv.org/abs/1812.05905.
[24] HAARNOJA T，ZHOU A，ABBEEL P，et al. Soft actor-critic:Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International Conference on Machine Learning. PMLR，2018:1861-1870.
[25] HOU Yuenan，LIU Lifeng，WEI Qing，et al. A novel DDPG method with prioritized experience replay[C]//2017 IEEE International Conference on Systems，Man，and Cybernetics (SMC). IEEE，2017:316-321.

基于时序差分学习模型预测控制的一体化自动驾驶换道策略

Integrated Autonomous Driving Lane Change Policy Based on Temporal Difference Learning Model Predictive Control

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	肖跃, 贺宜, 张鸣. 基于分层式MPC的分布式四轮驱动车辆轨迹跟踪控制方法[J]. 机械工程学报, 2026, 62(8): 475-488.
[2]	杨泽浩, 董威, 黄思翰, 阴艳超, 董李扬, 郑祖杰. 基于数字孪生和强化学习的智能生产线实时调度仿真优化方法[J]. 机械工程学报, 2026, 62(5): 12-25.
[3]	黄铭, 黄思翰, 陈建鹏, 董威, 王柏村, 阮兵, 高云鹏, 王国新, 阎艳. 基于多目标深度强化学习的不确定扰动下岛式装配线动态调度优化[J]. 机械工程学报, 2026, 62(5): 74-87.
[4]	陈其怀, 林添良, 马荣华, 温建和, 缪骋, 任好玲. 基于多模态的履带式挖掘机自动驾驶决策网络[J]. 机械工程学报, 2025, 61(4): 380-388.
[5]	胡炳涛, 钟锐锐, 冯毅雄, 杨晨, 王天跃, 洪兆溪, 谭建荣. 人-信息-物理互联环境下数字车间制造能力建模与自适应调度[J]. 机械工程学报, 2025, 61(3): 23-39.
[6]	姚福星, 李浩宇, 冷江昊, 杨雄基, 孙超. 面向多场景高速公路的多专家学习自主换道决策方法[J]. 机械工程学报, 2025, 61(22): 198-210.
[7]	杜国锋, 邵士博, 李尚霖, 林诚然, 曹政才. 融合视觉语言模型与近端策略优化算法的人形机器人步态切换方法[J]. 机械工程学报, 2025, 61(21): 204-212.
[8]	王国栋, 刘立, 孟宇, 白国星, 王家轩, 顾青. 融合摩擦因数估计的自动驾驶汽车集成路径跟踪控制[J]. 机械工程学报, 2025, 61(20): 243-252.
[9]	唐小林, 甘炯鹏, 张振果. 横纵向耦合跟车场景下基于多智能体深度强化学习的混合动力车队协同能量管理研究[J]. 机械工程学报, 2025, 61(2): 236-246.
[10]	林歆悠, 代军, 曾松榕. 基于运动风险的强化学习换道决策方法研究[J]. 机械工程学报, 2025, 61(18): 252-266.
[11]	曹延旭, 魏中宝, 宋若旸, 何洪文. 基于深度强化学习的跟驰场景中燃料电池混合动力汽车多目标能量管理研究[J]. 机械工程学报, 2025, 61(12): 191-203.
[12]	陈翔, 侯凯龙, 赵万忠, 王春燕. 考虑驾驶员特性的商用车电液复合转向系统自动驾驶转向模式切换控制[J]. 机械工程学报, 2025, 61(12): 204-216.
[13]	贺宜, 杜宇豪, 吴钪, 邱志军. 基于层次迭代搜索的自动驾驶重载半挂牵引车轨迹规划方法[J]. 机械工程学报, 2025, 61(10): 335-348.
[14]	李明富, 王飞鸿, 朱凌枫, 李想, 雷高攀, 刘翊, 李林凌, 侯宇葵, 胡余良. 接触丰富的自动装配方法研究进展：机器人装配接触力控制技能学习[J]. 机械工程学报, 2025, 61(10): 395-413.
[15]	王无印, 黄子钊, 庄子龙, 方怀瑾, 秦威. 基于深度强化学习的自动化码头堆场场桥调度方法[J]. 机械工程学报, 2024, 60(6): 44-57.