An Improved Q Learning Algorithm to Optimize Green Dynamic Scheduling Problem in a Reentrant Hybrid Flow Shop

doi:10.3901/JME.2023.13.246

Abstract

Abstract: Energy conservation and emission reduction are of great significance to achieve sustainable development. This study considers the influence of dynamic disturbance events on production and studies the reentrant hybrid flow shop green dynamic scheduling problem (RHFS-GD). An improved Q learning algorithm (IQL) is proposed to solve the RHFS-GD problem. In a reentrant hybrid flow shop, each stage is abstracted as an agent, and a multi-agent reinforcement learning model is established. The mean shift algorithm is used to cluster the historical states. To achieve global optimization, an experience sharing strategy is designed to realize the experience interaction among agents, and an adaptive greedy strategy is proposed to select actions. Finally, numerical experiments are carried out, and the experimental results show that the IQL algorithm is superior to single scheduling rules, which can improve production efficiency while ensuring low energy consumption, can quickly respond to dynamic disturbance events in the actual production, and can effectively solve practical problems.

Key words: energy conservation and emission reduction, reentrant hybrid flow shop, green dynamic scheduling, improved Q learning algorithm

CLC Number:

TP391

WU Xiuli, YAN Xiaoyan. An Improved Q Learning Algorithm to Optimize Green Dynamic Scheduling Problem in a Reentrant Hybrid Flow Shop[J]. Journal of Mechanical Engineering, 2023, 59(13): 246-259.

References

[1] MACHADO C G,WINROTH M P,RIBEIRO da SILVA E H D. Sustainable manufacturing in Industry 4.0:An emerging research agenda[J]. International Journal of Production Research,2020,58(5):1462-1484.
[2] 刘培基,刘飞,王旭,等. 绿色制造的理论与技术体系及其新框架[J]. 机械工程学报,2021,57(19):165-179. LIU Peiji,LIU Fei,WANG Xu,et al. The theory and technology system of green manufacturing and their new frameworks[J]. Journal of Mechanical Engineering,2021,57(19):165-179.
[3] 段建国,李豪晨,张青雷. 面向绿色制造的半组合式船用曲轴结构件生产车间多目标调度优化[J]. 计算机集成制造系统,2021,27(6):1714-1727. DUAN Jianguo,LI Haochen,ZHANG Qinglei. Green manufacturing-oriented multi-objective scheduling optimization for half built-up marine crank shaft component workshop[J]. Computer Integrated Manufacturing Systems,2021,27(6):1714-1727.
[4] 耿凯峰,叶春明,吴绍兴,等. 分时电价下多目标绿色可重入混合流水车间调度[J]. 中国机械工程,2020,31(12):1469-1480. GENG Kaifeng,YE Chunming,WU Shaoxing,et al. Multi-objective green re-entrant hybrid flow shop scheduling under time-of-use electricity tariffs[J]. China Mechanical Engineering,2020,31(12):1469-1480.
[5] MANSOURI S A,AKTAS E,BESIKCI U. Green scheduling of a two-machine flowshop:Trade-off between makespan and energy consumption[J]. European Journal of Operational Research,2016,248(3):772-788.
[6] KUMAR K P. Re-entrant lines[J]. Queueing System,1993,13(1-2):87-110.
[7] 吴秀丽,肖晓,赵宁. 考虑装卸的柔性作业车间双资源调度问题[J]. 控制与决策,2020,35(10):2475-2485. WU Xiuli,XIAO Xiao,ZHAO Ning. Flexible job shop dual resource scheduling problem considering loading and unloading[J]. Control and Decision,2020,35(10):2475-2485.
[8] 吴秀丽,崔琪. 考虑可再生能源的多目标柔性流水车间调度问题[J]. 计算机集成制造系统,2018,24(11):2792-2807. WU Xiuli,CUI Qi. Multi-objective flexible flow shop scheduling problem with renewable energy[J]. Computer Integrated Manufacturing Systems,2018,24(11):2792-2807.
[9] HUANG J D,LIU J J,CHEN Q X,et al. Minimizing makespan in a two-stage flow shop with parallel batch-processing machines and reentrant jobs[J]. Engineering Optimization,2019,49(6):1010-1023.
[10] ZHANG X Y,CHEN L. A re-entrant hybrid flow shop scheduling problem with machine eligibility constraints[J]. International Journal of Production Research,2018,56(16):5293-5305.
[11] 姚远远,叶春明,杨枫. 双目标可重入混合流水车间调度问题的离散灰狼优化算法[J]. 运筹与管理,2019,28(8):190-199. YAO Yuanyuan,YE Chunming,YANG Feng. Solving bi-objective reentrant hybrid flow shop scheduling problems by a hybrid discrete grey wolf optimizer[J]. Operations Research and Management Research,2019,28(8):190-199.
[12] 顾涛,李苏建,林莹璐,等. 周期式退火炉作批处理机的可重入批离散机流水车间调度[J]. 机械工程学报,2020,56(2):220-232. GU Tao,LI Sujian,LIN Yinglu,et al. Research on the re-entrant batch discrete flow shop scheduling for periodic annealing furnace as batch processor[J]. Journal of Mechanical Engineering,2020,56(2):220-232.
[13] YING K H,LIN S W,WAN S Y. Bi-objective reentrant hybrid flow shop scheduling:an iterated Pareto greedy algorithm[J]. International Journal of Production Research,2014,52(19):5735-5747.
[14] 赵梓焱,李思怡,刘士新,等. 钢铁生产过程动态调度综述[J]. 冶金自动化,2022,46(2):65-79. ZHAO Ziyan,LI Siyi,LIU Shixin,et al. Review on dynamic scheduling of steel production process[J]. Metallurgical Automation,2022,46(2):65-79.
[15] XIONG J,XING L,CHEN Y. Robust scheduling for multi-objective flexible job-shop problems with random machine breakdowns[J]. International Journal of Production Economics,2013,141(1):112-126.
[16] Al-HINAI N,ELMEKKAWY T. Robust and stable flexible job shop scheduling with random machine breakdowns using a hybrid genetic algorithm[J]. International Journal of Production Economics,2011,132(2):279-291.
[17] MEHTA S V,UZSOY R M. Predictable scheduling of a job shop subject to break-downs[J]. IEEE Robotics & Automation Magazine,1998,14(3):365-378.
[18] GAO K Z,SUGANTHAN P N,CHUA T J,et al. A two-stage artificial bee colony algorithm scheduling flexible job-shop scheduling problem with new job insertion[J]. Expert Systems with Applications,2015,42(21):7652-7663.
[19] NIE L,GAO L,LI P. Reactive scheduling in a job shop where jobs arrive over time[J]. Computers & Industrial Engineering,2013,66(2):389-405.
[20] 王维祺,叶春明,谭晓军. 基于Q学习算法的作业车间动态调度[J]. 计算机系统应用,2020,29(11):218-226. WANG Weiqi,YE Chunminge,TAN Xiaojun. Job shop dynamic scheduling based on Q-Learning algorithm[J]. Application of Computer System,2020,29(11):218-226.
[21] WANG L,PAN Z X,WANG J J. A review of reinforcement learning based intelligent optimization for manufacturing scheduling[J]. Complex System Modeling and Simulation,2021,1(4):257-270.
[22] ARVIV K,STERN H,EDAN Y. Collaborative reinforcement learning for a two-robot job transfer flowshop scheduling problem[J]. International journal of production research,2016,54(4):1196-1209.
[23] 曹红倩. 应用改进Q-learning算法解决柔性作业车间调度问题[J]. 国外电子测量技术,2022,41(4):164-169. CAO Hongqian. Application of improved Q-learning algorithm to solve flexible job shop scheduling problem[J]. Foreign Electronic Measurement Technology,2022,41(4):164-169.
[24] 韩忻辰,俞胜平,袁志明,等. 基于Q-learning的高速铁路列车动态调度方法[J]. 控制理论与应用,2021,38(10):1511-1521. HAN Xinchen,YU Shengping,YUAN Zhiming,et al. High-speed railway dynamic scheduling based on Q-learning method[J]. Control Theory and Applications,2021,38(10):1511-1521.
[25] 周芳芳,樊晓平,叶榛. 均值漂移算法的研究与应用[J].控制与决策,2007,22(8):841-847. ZHOU Fangfang,FAN Xiaoping,YE Zhen. Mean shift research and applications[J]. Control and Decision,2007,22(8):841-847.
[26] 杨能俊,郭宇,方伟光,等. 实时数据驱动的离散制造车间自适应调度方法[J]. 组合机床与自动化加工技术,2020,9(9):175-184. YANG Nengjun,GUO Yu,FANG Weiguang,et al. Real-time data driven adaptive scheduling method of discrete manufacturing workshops[J]. Modular Machine Tool & Automatic Manufacturing Technique,2020,9(9):175-184.