Real-time Energy Management Strategy for Hybrid Electric Vehicles Based on Deep Reinforcement Learning and Model Predictive Control

doi:10.3901/JME.260194

Abstract

Abstract: To address the challenge of balancing real-time performance and adaptability in hybrid electric vehicle (HEV) energy management, this paper proposes a real-time hierarchical energy management strategy (EMS) that integrates deep reinforcement learning (DRL) with model predictive control (MPC). At the upper layer, a deep Q-network (DQN) is employed to construct an EMS controller that rapidly plans a reference trajectory for the state of charge (SOC) prior to vehicle departure. At the lower level, a Long Short-Term Memory (LSTM) network is first employed to construct a velocity predictor, forecasting the velocity sequence over a future time domain. Subsequently, an MPC controller is designed to achieve optimal power flow allocation by tracking the SOC reference trajectory. The proposed strategy is then comprehensively compared with dynamic programming (DP) and rule-based strategies across different test conditions. Simulation results demonstrate that the proposed strategy achieves over 90% of the fuel economy attained by the DP strategy while exhibiting strong real-time application potential. Finally, hardware-in-the-loop (HIL) experiments validate the practical applicability of the proposed strategy.

Key words: hybrid electric vehicle, deep Q-network, model predictive control, energy management strategy

CLC Number:

U461

LIU Hui, Ma Xiaokang, HAN Lijin, XIANG Changle. Real-time Energy Management Strategy for Hybrid Electric Vehicles Based on Deep Reinforcement Learning and Model Predictive Control[J]. Journal of Mechanical Engineering, 2025, 62(6): 302-313.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

URL: http://www.cjmenet.com.cn/EN/10.3901/JME.260194

http://www.cjmenet.com.cn/EN/Y2025/V62/I6/302

References

[1] UROOJ A，NASIR A. Review of intelligent energy man agement techniques for hybrid electric vehicles [J]. Journal of Energy Storage，2024，92：112132.
[2] 郭景华，李文昌，王班，等. 基于深度强化学习的网联混合动力汽车队列控制[J]. 机械工程学报，2024，60(2)：262-271. GUO Jinhua，LI Wenchang，WANG Ban，et al. Queue control of connected hybrid electric vehicles based on deep reinforcement learning[J]. Journal of Mechanical Engineering，2024，60(2)：262-271.
[3] 解少博，屈鹏程，李嘉诚，等. 跟驰场景中网联混合电动货车速度规划和能量管理协同控制的研究[J]. 汽车工程，2022，44(8)：1136-1143，1152. XIE Shaobo，QU Pengcheng，LI Jiacheng，et al. A study on cooperative control of speed planning and energy management of grid-connected hybrid-electric trucks in a follow-mescenario[J]. Automotive Engineering，2022，44(8)：1136-1143，1152.
[4] RAMADAN H S，BECHERIF M，CLAUDE F. Energy management improvement of hybrid electric vehicles via combined GPS/rule-based methodology[J]. IEEE Transactions on Automation Science and Engineering， 2017，14(2)：586-597.
[5] BAGWE R M，BYERLY A，DOS SANTOS JR E C，et al. Adaptive rule-based energy management strategy for a parallel HEV[J]. Energies，2019，12(23)：4472.
[6] LI F，GAO L，ZHANG Y，et al. Hierarchical operation switch schedule algorithm for energy management strategy of hybrid electric vehicle using adaptive dynamic programming[J]. Sustainable Energy，Grids and Networks，2023，35：101107.
[7] TANG W，WANG Y，JIAO X，et al. Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios[J]. Energy，2023，265：126264.
[8] SHI D，LIU S，CAI Y，et al. Pontryagin’s minimum principle based fuzzy adaptive energy management for hybrid electric vehicle using real-time traffic information[J]. Applied Energy，2021，286：116467.
[9] KIM N，JEONG J，ZHENG C. Adaptive energy management strategy for plug-in hybrid electric vehicles with Pontryagin’s minimum principle based on daily driving patterns[J]. International Journal of Precision Engineering and Manufacturing-Green Technology，2019(6)：539-548.
[10] LI J，LIU Y，QIN D，et al. Research on equivalent factor boundary of equivalent consumption minimization strategy for PHEVs[J]. IEEE Transactions on Vehicular Technology，2020，69(6)：6011-6024.
[11] TIAN X，HE R，SUN X，et al. An ANFIS-based ECMS for energy optimization of parallel hybrid electric bus[J]. IEEE Transactions on Vehicular Technology，2019，69(2)：1473-1483.
[12] VIRGIN L，DAVIS R. Vibration isolation using buckled struts[J]. Journal of Sound Vibration，2003，260： 965-973.
[13] YANG N，RUAN S，HAN L，et al. Reinforcement learning-based real-time intelligent energy management for hybrid electric vehicles in a model predictive control framework[J]. Energy，2023，270：126971.
[14] GUO L，LIU H，HAN L，et al. Predictive energy management strategy of dual-mode hybrid electric vehicles combining dynamic coordination control and simultaneous power distribution[J]. Energy，2023，263：125598.
[15] GUO H，WANG X，LI L. State-of-charge-constraint-based energy management strategy of plug-in hybrid electric vehicle with bus route[J]. Energy Conversion and Management，2019，199：111972.
[16] LIN X，WU J，WEI Y. An ensemble learning velocity prediction-based energy management strategy for a plug-in hybrid electric vehicle considering driving pattern adaptive reference SOC[J]. Energy，2021，234：121308.
[17] ZHANG H，PENG J，TAN H，et al. Tackling SOC long-termdynamic for energy management of hybrid electric buses via adaptive policy optimization[J]. Applied energy，2020，269：115031.
[18] 杨宁康，韩立金，刘辉，等. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程，2021，43(7)：1046-1056. YANG Ningkang，HAN Lijin，LIU Hui，et al. Research on enhanced learning energy management strategy for hybrid vehicles based on efficiency optimization[J]. Automotive Engineering，2021，43(7)：1046-1056.
[19] HAN X，HE H，WU J，et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy，2019，254：113708.
[20] LI W，CUI H，NEMETH T，et al. Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles[J]. Journal of Energy Storage，2021，36：102355.
[21] SHI D，XU H，WANG S，et al. Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network[J]. Energy，2024，305：132402.
[22] 胡晓松，陈科坪，唐小林，等. 基于机器学习速度预测的并联混合动力车辆能量管理研究[J]. 机械工程学报，2020，56(16)：181-192. HU Xiaosong，CHEN Keping，TANG Xiaolin，et al. Research on energy management of parallel hybrid electric vehicles based on machine learning speed prediction[J]. Journal of Mechanical Engineering，2020，56(16)：181-192.
[23] 唐小林，陈佳信，刘腾，等. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究[J]. 机械工程学报，2021，57(22)：237-246. TANG Xiaolin，CHEN Jiaxin，LIU Teng，et al. Research on intelligent following control and energy management strategy of hybrid electric vehicles based on deep reinforcement learning[J]. Journal of Mechanical Engineering，2021，57(22)：237-246.