基于深度强化学习和模型预测控制的混合动力电动汽车实时能量管理策略

doi:10.3901/JME.260194

摘要/Abstract

摘要： 为应对混合动力电动汽车(Hybrid electric vehicle，HEV)能量管理在实时性和适应性之间难以协调的挑战，提出一种结合深度强化学习(Deep reinforcement learning, DRL)和模型预测控制(Model predictive control, MPC)的实时分层能量管理策略(Energy management strategy, EMS)。在上层，利用深度Q网络(Deep Q-network, DQN)构建一种能量管理策略控制器，以便在汽车出发前快速规划电池荷电状态(State of charge, SOC)参考轨迹。在下层，首先利用长短期记忆网络(Long short-term memory, LSTM)构建一种速度预测器，进行未来一定时域范围内的速度序列预测，接着设计一个MPC控制器通过跟踪SOC参考轨迹完成最优功率流分配；然后将所提策略与动态规划(Dynamic programming，DP)、基于规则的策略在不同测试工况下进行综合性能比较，仿真结果表明，所提策略的燃料经济性达到了DP策略的90%以上，同时展现出了良好的实时应用潜力，最后通过硬件在环(Hardware-in-the-Loop，HIL)试验验证了所提策略在实际应用中的潜力。

关键词: 混合电动汽车, 深度Q网络, 模型预测控制, 能量管理策略

Abstract: To address the challenge of balancing real-time performance and adaptability in hybrid electric vehicle (HEV) energy management, this paper proposes a real-time hierarchical energy management strategy (EMS) that integrates deep reinforcement learning (DRL) with model predictive control (MPC). At the upper layer, a deep Q-network (DQN) is employed to construct an EMS controller that rapidly plans a reference trajectory for the state of charge (SOC) prior to vehicle departure. At the lower level, a Long Short-Term Memory (LSTM) network is first employed to construct a velocity predictor, forecasting the velocity sequence over a future time domain. Subsequently, an MPC controller is designed to achieve optimal power flow allocation by tracking the SOC reference trajectory. The proposed strategy is then comprehensively compared with dynamic programming (DP) and rule-based strategies across different test conditions. Simulation results demonstrate that the proposed strategy achieves over 90% of the fuel economy attained by the DP strategy while exhibiting strong real-time application potential. Finally, hardware-in-the-loop (HIL) experiments validate the practical applicability of the proposed strategy.

Key words: hybrid electric vehicle, deep Q-network, model predictive control, energy management strategy

中图分类号:

U461

刘辉, 马小康, 韩立金, 项昌乐. 基于深度强化学习和模型预测控制的混合动力电动汽车实时能量管理策略[J]. 机械工程学报, 2025, 62(6): 302-313.

LIU Hui, Ma Xiaokang, HAN Lijin, XIANG Changle. Real-time Energy Management Strategy for Hybrid Electric Vehicles Based on Deep Reinforcement Learning and Model Predictive Control[J]. Journal of Mechanical Engineering, 2025, 62(6): 302-313.

导出引用管理器 EndNote|Reference Manager|ProCite|BibTeX|RefWorks

链接本文: http://www.cjmenet.com.cn/CN/10.3901/JME.260194

http://www.cjmenet.com.cn/CN/Y2025/V62/I6/302

参考文献

[1] UROOJ A，NASIR A. Review of intelligent energy man agement techniques for hybrid electric vehicles [J]. Journal of Energy Storage，2024，92：112132.
[2] 郭景华，李文昌，王班，等. 基于深度强化学习的网联混合动力汽车队列控制[J]. 机械工程学报，2024，60(2)：262-271. GUO Jinhua，LI Wenchang，WANG Ban，et al. Queue control of connected hybrid electric vehicles based on deep reinforcement learning[J]. Journal of Mechanical Engineering，2024，60(2)：262-271.
[3] 解少博，屈鹏程，李嘉诚，等. 跟驰场景中网联混合电动货车速度规划和能量管理协同控制的研究[J]. 汽车工程，2022，44(8)：1136-1143，1152. XIE Shaobo，QU Pengcheng，LI Jiacheng，et al. A study on cooperative control of speed planning and energy management of grid-connected hybrid-electric trucks in a follow-mescenario[J]. Automotive Engineering，2022，44(8)：1136-1143，1152.
[4] RAMADAN H S，BECHERIF M，CLAUDE F. Energy management improvement of hybrid electric vehicles via combined GPS/rule-based methodology[J]. IEEE Transactions on Automation Science and Engineering， 2017，14(2)：586-597.
[5] BAGWE R M，BYERLY A，DOS SANTOS JR E C，et al. Adaptive rule-based energy management strategy for a parallel HEV[J]. Energies，2019，12(23)：4472.
[6] LI F，GAO L，ZHANG Y，et al. Hierarchical operation switch schedule algorithm for energy management strategy of hybrid electric vehicle using adaptive dynamic programming[J]. Sustainable Energy，Grids and Networks，2023，35：101107.
[7] TANG W，WANG Y，JIAO X，et al. Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios[J]. Energy，2023，265：126264.
[8] SHI D，LIU S，CAI Y，et al. Pontryagin’s minimum principle based fuzzy adaptive energy management for hybrid electric vehicle using real-time traffic information[J]. Applied Energy，2021，286：116467.
[9] KIM N，JEONG J，ZHENG C. Adaptive energy management strategy for plug-in hybrid electric vehicles with Pontryagin’s minimum principle based on daily driving patterns[J]. International Journal of Precision Engineering and Manufacturing-Green Technology，2019(6)：539-548.
[10] LI J，LIU Y，QIN D，et al. Research on equivalent factor boundary of equivalent consumption minimization strategy for PHEVs[J]. IEEE Transactions on Vehicular Technology，2020，69(6)：6011-6024.
[11] TIAN X，HE R，SUN X，et al. An ANFIS-based ECMS for energy optimization of parallel hybrid electric bus[J]. IEEE Transactions on Vehicular Technology，2019，69(2)：1473-1483.
[12] VIRGIN L，DAVIS R. Vibration isolation using buckled struts[J]. Journal of Sound Vibration，2003，260： 965-973.
[13] YANG N，RUAN S，HAN L，et al. Reinforcement learning-based real-time intelligent energy management for hybrid electric vehicles in a model predictive control framework[J]. Energy，2023，270：126971.
[14] GUO L，LIU H，HAN L，et al. Predictive energy management strategy of dual-mode hybrid electric vehicles combining dynamic coordination control and simultaneous power distribution[J]. Energy，2023，263：125598.
[15] GUO H，WANG X，LI L. State-of-charge-constraint-based energy management strategy of plug-in hybrid electric vehicle with bus route[J]. Energy Conversion and Management，2019，199：111972.
[16] LIN X，WU J，WEI Y. An ensemble learning velocity prediction-based energy management strategy for a plug-in hybrid electric vehicle considering driving pattern adaptive reference SOC[J]. Energy，2021，234：121308.
[17] ZHANG H，PENG J，TAN H，et al. Tackling SOC long-termdynamic for energy management of hybrid electric buses via adaptive policy optimization[J]. Applied energy，2020，269：115031.
[18] 杨宁康，韩立金，刘辉，等. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程，2021，43(7)：1046-1056. YANG Ningkang，HAN Lijin，LIU Hui，et al. Research on enhanced learning energy management strategy for hybrid vehicles based on efficiency optimization[J]. Automotive Engineering，2021，43(7)：1046-1056.
[19] HAN X，HE H，WU J，et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy，2019，254：113708.
[20] LI W，CUI H，NEMETH T，et al. Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles[J]. Journal of Energy Storage，2021，36：102355.
[21] SHI D，XU H，WANG S，et al. Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network[J]. Energy，2024，305：132402.
[22] 胡晓松，陈科坪，唐小林，等. 基于机器学习速度预测的并联混合动力车辆能量管理研究[J]. 机械工程学报，2020，56(16)：181-192. HU Xiaosong，CHEN Keping，TANG Xiaolin，et al. Research on energy management of parallel hybrid electric vehicles based on machine learning speed prediction[J]. Journal of Mechanical Engineering，2020，56(16)：181-192.
[23] 唐小林，陈佳信，刘腾，等. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究[J]. 机械工程学报，2021，57(22)：237-246. TANG Xiaolin，CHEN Jiaxin，LIU Teng，et al. Research on intelligent following control and energy management strategy of hybrid electric vehicles based on deep reinforcement learning[J]. Journal of Mechanical Engineering，2021，57(22)：237-246.