基于云控系统高精度地图驱动的深度强化学习型混合动力汽车集成控制

doi:10.3901/JME.2022.24.163

机械工程学报 ›› 2022, Vol. 58 ›› Issue (24): 163-177.doi: 10.3901/JME.2022.24.163

扫码分享

基于云控系统高精度地图驱动的深度强化学习型混合动力汽车集成控制

唐小林¹, 陈佳信¹, 高博麟², 杨凯¹, 胡晓松¹, 李克强²

1. 重庆大学机械与运载工程学院重庆 400044;
2. 清华大学汽车安全与节能国家重点实验室北京 100084

收稿日期:2022-03-09 修回日期:2022-07-15 出版日期:2022-12-20 发布日期:2023-04-03
通讯作者: 高博麟(通信作者),男,1986年出生,博士,副研究员。主要研究方向为智能网联驾驶系统动态设计与控制、车路云一体化协同感知、决策和控制。E-mail:gaobolin@tsinghua.edu.cn
作者简介:唐小林,男,1984年出生,博士,副教授,博士研究生导师。主要研究方向为混合动力汽车动力学与节能控制。E-mail:tangxl0923@cqu.edu.cn;陈佳信,男,1996年出生,硕士研究生。主要研究方向为混合动力汽车能量管理与深度强化学习。E-mail:201932132050@cqu.edu.cn;高博麟(通信作者),男,1986年出生,博士,副研究员。主要研究方向为智能网联驾驶系统动态设计与控制、车路云一体化协同感知、决策和控制。E-mail:gaobolin@tsinghua.edu.cn
基金资助:
国家自然科学基金资助项目（52222215，52072051）。

Deep Reinforcement Learning-based Integrated Control of Hybrid Electric Vehicles Driven by High Definition Map in Cloud Control System

TANG Xiaolin¹, CHEN Jiaxin¹, GAO Bolin², YANG Kai¹, HU Xiaosong¹, LI Keqiang²

1. College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044;
2. State Key Laboratory of Automotive Safety and Energy, Tsinghua University, Beijing 100084

Received:2022-03-09 Revised:2022-07-15 Online:2022-12-20 Published:2023-04-03

摘要/Abstract

摘要： 在智能化、网联化与新能源化的发展背景下，汽车工业将联合计算机、信息通信、人工智能等领域实现融合性发展。基于新一代信息与通信技术——智能网联汽车云控系统，通过网联数据驱动的形式实现新能源汽车的云控级自动驾驶，将为车辆行驶与动力系统提供革新的规划与控制思路。首先，基于云控系统的资源平台获取目标路段的经纬度、海拔、气象信息，建立包含坡度、曲率、转角等数据在内的高精度模型。其次，提出了一种基于高精地图驱动的深度强化学习型混合动力汽车集成控制方法，通过利用两种深度强化学习算法对整车层的速度与转向以及动力系统层的发动机与变速器进行控制，实现了四种控制策略的同步学习。最后，采用高性能边缘计算设备NVIDIA Jetson AGX Xavier进行了处理器在环测试。结果表明，当变量空间涉及14种状态与4种动作时，深度强化学习型集成控制策略在全程172 km的高速工况下实现了在整车层对速度与转向的精准控制，同时取得了5.53 L/100 km的燃油经济性，并且在嵌入式处理器在环测试中仅消耗104.14 s的计算时间，有效验证了学习型多目标集成控制策略的优化性与实时性。

关键词: 云控系统, 高精地图, 深度强化学习, 混合动力汽车, 集成控制

Abstract: In the context of the development of intelligence, connectivity, and new energy, the automotive industry combines computer, information communication, artificial intelligence(AI) to achieve integrated development. Based on the new generation of information and communication technology——cloud control system(CCS) of intelligent and connected vehicles(ICVs), the cloud-level automatic driving of new energy vehicles is realized driven by connected data, which provides innovative planning and control ideas for vehicle driving and power systems. Firstly, based on the resource platform of CCS, the latitude, longitude, altitude, and weather of the target road are obtained, and a high definition(HD) path model including slope, curvature, and steering angle is established. Secondly, a deep reinforcement learning(DRL)-based integrated control method for hybrid electric vehicle(HEV) drive by the HD model is proposed. By adopting two DRL algorithms, the speed and steering of the vehicle and the engine and transmission in the powertrain are controlled, and the synchronous learning of four control strategies is realized. Finally, processor-in-the-loop(PIL) tests are performed by using the high-performance edge computing device NVIDIA Jetson AGX Xavier. The results show that under a variable space including 14 states and 4 actions, the DRL -based integrated control strategy realizes the precise control of the speed and steering of the vehicle layer under the high-speed driving cycle of 172 km, and achieves a fuel consumption of 5.53L/100km. Meanwhile, it only consumes 104.14s in the PIL test, which verifies the optimization and real-time performance of the learning-based multi-objective integrated control strategy.

Key words: cloud control system, high definition map, deep reinforcement learning, hybrid electric vehicle, integrated control

中图分类号:

U461

唐小林, 陈佳信, 高博麟, 杨凯, 胡晓松, 李克强. 基于云控系统高精度地图驱动的深度强化学习型混合动力汽车集成控制[J]. 机械工程学报, 2022, 58(24): 163-177.

TANG Xiaolin, CHEN Jiaxin, GAO Bolin, YANG Kai, HU Xiaosong, LI Keqiang. Deep Reinforcement Learning-based Integrated Control of Hybrid Electric Vehicles Driven by High Definition Map in Cloud Control System[J]. Journal of Mechanical Engineering, 2022, 58(24): 163-177.

参考文献

[1] 李克强, 戴一凡, 李升波, 等.智能网联汽车(ICV)技术的发展现状及趋势[J].汽车安全与节能学报, 2017, 8(1):1-14.LI Keqiang, DAI Yifan, LI Shengbo, et al.State-of-the-art and technical trends of intelligent and connected vehicles[J].Journal of Automotive Safety and Energy, 2017, 8(1):1-14.
[2] 李克强, 常雪阳, 李家文, 等.智能网联汽车云控系统及其实现[J].汽车工程, 2020, 42(12):1595-1605.LI Keqiang, CHANG Xueyang, LI Jiawen, et al.Cloud control system for intelligent and connected vehicles and its application[J].Automotive Engineering, 2020, 42(12):1595-1605.
[3] 欧阳明高.中国新能源汽车的研发及展望[J].科技导报, 2016, 34(6):13-20.OUYANG Minggao.New energy vehicle research and development in China[J].Science and Technology Review, 2016, 34(6):13-20.
[4] TANG X L, JIA T, HU X S, et al.Naturalistic data-driven predictive energy management for plug-in hybrid electric vehicles[J].IEEE Transactions on Transportation Electrification, 2021, 7(2):497-508.
[5] TANG X L, CHEN J X, PU H Y, et al.Double deep reinforcement learning-based energy management for a parallel hybrid electric vehicle with engine start-stop strategy[J].IEEE Transactions on Transportation Electrification, 2022, 8(1):1376-1388.
[6] 刘华伟, 耿安琪, 何正友, 等.重载铁路再生制动能量利用方案研究[J].电气工程学报, 2021, 16(1):157-165.LIU Huawei, GENG Anqi, HE Zhengyou, et al.Research on energy utilization scheme of regenerative braking for heavy haul railway[J].Journal of Electrical Engineering, 2021, 16(1):157-165.
[7] 肖梓林.城市轨道交通再生能量利用的直流牵引供电系统仿真研究[J].电气工程学报, 2021, 16(1):166-172.XIAO Zilin.Simulation research on DC traction power supply system for renewable energy utilization of urban rail transit[J].Journal of Electrical Engineering, 2021, 16(1):166-172.
[8] TRAN DD, VAFAEIPOUR M, EL BAGHDADI, M, et al.Thorough state-of-the-art analysis of electric and hybrid vehicle powertrains:Topologies and integrated energy management strategies[J].Renewable & Sustainable Energy Reviews, 2020, 119:109596.
[9] 张风奇, 胡晓松, 许康辉, 等.混合动力汽车模型预测能量管理研究现状与展望[J].机械工程学报, 2019, 55(10):86-108.ZHANG Fengqi, HU Xiaosong, XU Kanghui, et al.Current status and prospects for model predictive energy management in hybrid electric vehicles[J].Journal of Mechanical Engineering, 2019, 55(10):86-108.
[10] YANG K, TANG X L, QIN Y C, et al.Comparative study of trajectory tracking control for automated vehicles via model predictive control and robust H-infinity state feedback control[J].Chinese Journal of Mechanical Engineering, 2021, 34(1):1-14.
[11] LIU T, HU X S, LI S B, et al.Reinforcement learning optimized look-ahead energy management of a parallel hybrid electric vehicle[J].IEEE-ASME Transactions on Mechatronics, 2017, 22(4):1497-1507.
[12] HU X S, LIU T, QI X W, et al.Reinforcement learning for hybrid and plug-in hybrid electric vehicle energy management:Recent advances and prospects[J].IEEE Industrial Electronics Magazine, 2019, 13(3):16-25.
[13] TAN H C, ZHANG H L, PENG J K, et al.Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space[J].Energy Conversion and Management, 2019, 195:548-560.
[14] ZOU R N, FAN L K, DONG Y R, et al.DQL energy management:An online-updated algorithm and its application in fix-line hybrid electric vehicle[J].Energy, 2021, 225:120174.
[15] LI Y C, HE H W, KHAJEPOUR A, et al.Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information[J].Applied Energy, 2019(255):113762.
[16] WANG Y, TAN H C, WU Y K, et al.Hybrid electric vehicle energy management with computer vision and deep reinforcement learning[J].IEEE Transactions on Industrial Informatics, 2021, 17(6):3857-3868.
[17] LI W H, CUI H, NEMETH T, et al.Cloud-based health-conscious energy management of hybrid battery systems in electric vehicles with deep reinforcement learning[J].Applied Energy, 2021, 293:116977.
[18] LIAN R Z, TAN H C, PENG J K, et al.Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management[J].IEEE Transactions on Vehicular Technology, 2020, 69(8):8367-8380.
[19] TANG X L, CHEN J X, LIU T, et al.Distributed deep reinforcement learning-based energy and emission management strategy for hybrid electric vehicles[J].IEEE Transactions on Vehicular Technology, 2021, 70(10):9922-9934.
[20] 李克强, 李家文, 常雪阳, 等.智能网联汽车云控系统原理及其典型应用[J].汽车安全与节能学报, 2020, 11(3):261-275.LI Keqiang, LI Jiawen, CHANG Xueyang, et al.Principles and typical applications of cloud control system for intelligent and connected vehicles[J].Journal of Automotive Safety and Energy, 2020, 11(3):261-275.
[21] 唐小林, 李珊珊, 王红, 等.网联环境下基于分层式模型预测控制的车队能量控制策略研究[J].机械工程学报, 2020, 56(14):119-128.TANG Xiaolin, LI Shanshan, WANG Hong, et al.Research on energy control strategy based on hierarchical model predictive control in connected environment[J].Journal of Mechanical Engineering, 2020, 56(14):119-128.
[22] 唐小林, 陈佳信, 刘腾, 等.基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究[J].机械工程学报, 2021, 57(22):237-246.TANG Xiaolin, CHEN Jiaxin, LIU Teng, et al.Research on deep reinforcement learning-based intelligent car-following control and energy management strategy for hybrid electric vehicles[J].Journal of Mechanical Engineering, 2021, 57(22):237-246.
[23] 刘腾.混合动力车辆强化学习能量管理研究[D].北京:北京理工大学, 2017.LIU Teng.Reinforcement learning-based energy management for hybrid electric vehicles[D].Beijing:Beijing Institute of Technology, 2017.
[24] CHEN J X, SHU H, TANG X L, et al.Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment[J].Energy, 2022, 239, Part C:122123.
[25] VOLODYMYR M, KORAY K, DAVID S, et al.Human-level control through deep reinforcement learning[J].Nature, 2015, 518(7540):529-533.
[26] 胡悦.混合动力电动汽车控制系统设计与能量管理策略研究[D].深圳:中国科学院大学(中国科学院深圳先进技术研究院), 2018.HU Yue.Research on control system design and energy management strategy of hybrid electric vehicle[D].Shenzhen:University of Chinese Academy of Sciences (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences), 2018.

基于云控系统高精度地图驱动的深度强化学习型混合动力汽车集成控制

Deep Reinforcement Learning-based Integrated Control of Hybrid Electric Vehicles Driven by High Definition Map in Cloud Control System

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王无印, 黄子钊, 庄子龙, 方怀瑾, 秦威. 基于深度强化学习的自动化码头堆场场桥调度方法[J]. 机械工程学报, 2024, 60(6): 44-57.
[2]	郭景华, 李文昌, 王班, 王靖瑶. 基于深度强化学习的网联混合动力汽车队列控制[J]. 机械工程学报, 2024, 60(2): 262-271.
[3]	张利鹏, 王子健, 任长安, 樊小建. 混合动力汽车增速离合器创新设计与应用研究[J]. 机械工程学报, 2023, 59(8): 253-263.
[4]	钟沛成, 骆德渊, 庞明君. 基于深度强化学习的四足机器人跟随策略研究及系统实现[J]. 机械工程学报, 2023, 59(13): 79-88.
[5]	陈睿奇, 黎雯馨, 王传洋, 杨宏兵. 基于深度强化学习的工序交互式智能体Job shop调度方法[J]. 机械工程学报, 2023, 59(12): 78-88.
[6]	唐鑫, 欧阳权, 黄俍卉, 王志胜, 马瑞. 基于深度强化学习的锂电池快速充电控制策略[J]. 机械工程学报, 2022, 58(22): 69-78.
[7]	王辉, 徐佳文, 严如强. 基于多尺度注意力深度强化学习网络的行星齿轮箱智能诊断方法[J]. 机械工程学报, 2022, 58(11): 133-142.
[8]	唐小林, 陈佳信, 刘腾, 李佳承, 胡晓松. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究[J]. 机械工程学报, 2021, 57(22): 237-246.
[9]	胡建军, 刘子睿, 梅博, 彭航. 一种单行星排功率分流式混合动力系统构型优化设计方案[J]. 机械工程学报, 2021, 57(18): 264-276.
[10]	陈超逸, 鲁娟, 陈楷, 黎宇嘉, 马俊燕, 廖小平. 车削表面粗糙度解析模型与DDQN-SVR预测模型研究[J]. 机械工程学报, 2021, 57(13): 262-272.
[11]	施群, 吕雷, 谢家骏. 可变环境下仿人机器人智能姿态控制[J]. 机械工程学报, 2020, 56(3): 64-72.
[12]	郭鹏, 张新艳, 余建波. 基于深度强化学习与有限元仿真集成的拉深成形控制[J]. 机械工程学报, 2020, 56(20): 47-58.
[13]	胡晓松, 陈科坪, 唐小林, 王斌. 基于机器学习速度预测的并联混合动力车辆能量管理研究[J]. 机械工程学报, 2020, 56(16): 181-192.
[14]	唐小林, 李珊珊, 王红, 段紫文, 李以农, 郑玲. 网联环境下基于分层式模型预测控制的车队能量控制策略研究[J]. 机械工程学报, 2020, 56(14): 119-128.
[15]	刘辉, 李训明, 王伟达, 韩立金, 闫正军. 基于最优功率分配因子的插电式混合动力汽车实时能量管理策略研究[J]. 机械工程学报, 2019, 55(4): 91-101.