基于改进Q学习的可重入混合流水车间绿色动态调度

doi:10.3901/JME.2023.13.246

机械工程学报 ›› 2023, Vol. 59 ›› Issue (13): 246-259.doi: 10.3901/JME.2023.13.246

扫码分享

基于改进Q学习的可重入混合流水车间绿色动态调度

吴秀丽, 闫晓燕

北京科技大学机械工程学院北京 100083

收稿日期:2022-07-09 修回日期:2022-12-19 出版日期:2023-07-05 发布日期:2023-08-15
通讯作者: 吴秀丽(通信作者),女,1977年出生,博士,教授,硕士研究生导师。主要研究方向为生产计划与调度、机器学习和智能算法。E-mail:wuxiuli@ustb.edu.cn
作者简介:闫晓燕,女,1997年出生。主要研究方向为生产计划与调度、机器学习和智能算法。E-mail:18148252183@163.com
基金资助:
国家自然科学基金资助项目(52175499)。

An Improved Q Learning Algorithm to Optimize Green Dynamic Scheduling Problem in a Reentrant Hybrid Flow Shop

WU Xiuli, YAN Xiaoyan

School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083

Received:2022-07-09 Revised:2022-12-19 Online:2023-07-05 Published:2023-08-15

摘要/Abstract

摘要： 节能减排对于实现可持续发展具有重要意义。考虑了动态扰动事件对生产的影响，研究了可重入混合流水车间绿色动态调度问题，提出了改进的Q学习算法。在可重入混合流水车间中，将各个加工阶段抽象为智能体，搭建了多智能体强化学习模型。选用均值漂移算法对历史状态进行聚类。为实现全局优化，设计了经验共享策略实现各个智能体之间的经验交互，并设计了自适应贪婪策略选取动作。最后进行了数值实验，实验结果表明，在求解可重入混合流水车间绿色动态调度问题时，改进的Q学习算法优于单一的调度规则，可以在提高生产效率的同时保证较低的能耗，并且能够对实际生产环境中的动态扰动因素快速做出反应，能够有效地解决实际问题。

关键词: 节能减排, 可重入混合流水车间, 绿色动态调度, 改进的Q学习算法

Abstract: Energy conservation and emission reduction are of great significance to achieve sustainable development. This study considers the influence of dynamic disturbance events on production and studies the reentrant hybrid flow shop green dynamic scheduling problem (RHFS-GD). An improved Q learning algorithm (IQL) is proposed to solve the RHFS-GD problem. In a reentrant hybrid flow shop, each stage is abstracted as an agent, and a multi-agent reinforcement learning model is established. The mean shift algorithm is used to cluster the historical states. To achieve global optimization, an experience sharing strategy is designed to realize the experience interaction among agents, and an adaptive greedy strategy is proposed to select actions. Finally, numerical experiments are carried out, and the experimental results show that the IQL algorithm is superior to single scheduling rules, which can improve production efficiency while ensuring low energy consumption, can quickly respond to dynamic disturbance events in the actual production, and can effectively solve practical problems.

Key words: energy conservation and emission reduction, reentrant hybrid flow shop, green dynamic scheduling, improved Q learning algorithm

中图分类号:

TP391

吴秀丽, 闫晓燕. 基于改进Q学习的可重入混合流水车间绿色动态调度[J]. 机械工程学报, 2023, 59(13): 246-259.

WU Xiuli, YAN Xiaoyan. An Improved Q Learning Algorithm to Optimize Green Dynamic Scheduling Problem in a Reentrant Hybrid Flow Shop[J]. Journal of Mechanical Engineering, 2023, 59(13): 246-259.

参考文献

[1] MACHADO C G,WINROTH M P,RIBEIRO da SILVA E H D. Sustainable manufacturing in Industry 4.0:An emerging research agenda[J]. International Journal of Production Research,2020,58(5):1462-1484.
[2] 刘培基,刘飞,王旭,等. 绿色制造的理论与技术体系及其新框架[J]. 机械工程学报,2021,57(19):165-179. LIU Peiji,LIU Fei,WANG Xu,et al. The theory and technology system of green manufacturing and their new frameworks[J]. Journal of Mechanical Engineering,2021,57(19):165-179.
[3] 段建国,李豪晨,张青雷. 面向绿色制造的半组合式船用曲轴结构件生产车间多目标调度优化[J]. 计算机集成制造系统,2021,27(6):1714-1727. DUAN Jianguo,LI Haochen,ZHANG Qinglei. Green manufacturing-oriented multi-objective scheduling optimization for half built-up marine crank shaft component workshop[J]. Computer Integrated Manufacturing Systems,2021,27(6):1714-1727.
[4] 耿凯峰,叶春明,吴绍兴,等. 分时电价下多目标绿色可重入混合流水车间调度[J]. 中国机械工程,2020,31(12):1469-1480. GENG Kaifeng,YE Chunming,WU Shaoxing,et al. Multi-objective green re-entrant hybrid flow shop scheduling under time-of-use electricity tariffs[J]. China Mechanical Engineering,2020,31(12):1469-1480.
[5] MANSOURI S A,AKTAS E,BESIKCI U. Green scheduling of a two-machine flowshop:Trade-off between makespan and energy consumption[J]. European Journal of Operational Research,2016,248(3):772-788.
[6] KUMAR K P. Re-entrant lines[J]. Queueing System,1993,13(1-2):87-110.
[7] 吴秀丽,肖晓,赵宁. 考虑装卸的柔性作业车间双资源调度问题[J]. 控制与决策,2020,35(10):2475-2485. WU Xiuli,XIAO Xiao,ZHAO Ning. Flexible job shop dual resource scheduling problem considering loading and unloading[J]. Control and Decision,2020,35(10):2475-2485.
[8] 吴秀丽,崔琪. 考虑可再生能源的多目标柔性流水车间调度问题[J]. 计算机集成制造系统,2018,24(11):2792-2807. WU Xiuli,CUI Qi. Multi-objective flexible flow shop scheduling problem with renewable energy[J]. Computer Integrated Manufacturing Systems,2018,24(11):2792-2807.
[9] HUANG J D,LIU J J,CHEN Q X,et al. Minimizing makespan in a two-stage flow shop with parallel batch-processing machines and reentrant jobs[J]. Engineering Optimization,2019,49(6):1010-1023.
[10] ZHANG X Y,CHEN L. A re-entrant hybrid flow shop scheduling problem with machine eligibility constraints[J]. International Journal of Production Research,2018,56(16):5293-5305.
[11] 姚远远,叶春明,杨枫. 双目标可重入混合流水车间调度问题的离散灰狼优化算法[J]. 运筹与管理,2019,28(8):190-199. YAO Yuanyuan,YE Chunming,YANG Feng. Solving bi-objective reentrant hybrid flow shop scheduling problems by a hybrid discrete grey wolf optimizer[J]. Operations Research and Management Research,2019,28(8):190-199.
[12] 顾涛,李苏建,林莹璐,等. 周期式退火炉作批处理机的可重入批离散机流水车间调度[J]. 机械工程学报,2020,56(2):220-232. GU Tao,LI Sujian,LIN Yinglu,et al. Research on the re-entrant batch discrete flow shop scheduling for periodic annealing furnace as batch processor[J]. Journal of Mechanical Engineering,2020,56(2):220-232.
[13] YING K H,LIN S W,WAN S Y. Bi-objective reentrant hybrid flow shop scheduling:an iterated Pareto greedy algorithm[J]. International Journal of Production Research,2014,52(19):5735-5747.
[14] 赵梓焱,李思怡,刘士新,等. 钢铁生产过程动态调度综述[J]. 冶金自动化,2022,46(2):65-79. ZHAO Ziyan,LI Siyi,LIU Shixin,et al. Review on dynamic scheduling of steel production process[J]. Metallurgical Automation,2022,46(2):65-79.
[15] XIONG J,XING L,CHEN Y. Robust scheduling for multi-objective flexible job-shop problems with random machine breakdowns[J]. International Journal of Production Economics,2013,141(1):112-126.
[16] Al-HINAI N,ELMEKKAWY T. Robust and stable flexible job shop scheduling with random machine breakdowns using a hybrid genetic algorithm[J]. International Journal of Production Economics,2011,132(2):279-291.
[17] MEHTA S V,UZSOY R M. Predictable scheduling of a job shop subject to break-downs[J]. IEEE Robotics & Automation Magazine,1998,14(3):365-378.
[18] GAO K Z,SUGANTHAN P N,CHUA T J,et al. A two-stage artificial bee colony algorithm scheduling flexible job-shop scheduling problem with new job insertion[J]. Expert Systems with Applications,2015,42(21):7652-7663.
[19] NIE L,GAO L,LI P. Reactive scheduling in a job shop where jobs arrive over time[J]. Computers & Industrial Engineering,2013,66(2):389-405.
[20] 王维祺,叶春明,谭晓军. 基于Q学习算法的作业车间动态调度[J]. 计算机系统应用,2020,29(11):218-226. WANG Weiqi,YE Chunminge,TAN Xiaojun. Job shop dynamic scheduling based on Q-Learning algorithm[J]. Application of Computer System,2020,29(11):218-226.
[21] WANG L,PAN Z X,WANG J J. A review of reinforcement learning based intelligent optimization for manufacturing scheduling[J]. Complex System Modeling and Simulation,2021,1(4):257-270.
[22] ARVIV K,STERN H,EDAN Y. Collaborative reinforcement learning for a two-robot job transfer flowshop scheduling problem[J]. International journal of production research,2016,54(4):1196-1209.
[23] 曹红倩. 应用改进Q-learning算法解决柔性作业车间调度问题[J]. 国外电子测量技术,2022,41(4):164-169. CAO Hongqian. Application of improved Q-learning algorithm to solve flexible job shop scheduling problem[J]. Foreign Electronic Measurement Technology,2022,41(4):164-169.
[24] 韩忻辰,俞胜平,袁志明,等. 基于Q-learning的高速铁路列车动态调度方法[J]. 控制理论与应用,2021,38(10):1511-1521. HAN Xinchen,YU Shengping,YUAN Zhiming,et al. High-speed railway dynamic scheduling based on Q-learning method[J]. Control Theory and Applications,2021,38(10):1511-1521.
[25] 周芳芳,樊晓平,叶榛. 均值漂移算法的研究与应用[J].控制与决策,2007,22(8):841-847. ZHOU Fangfang,FAN Xiaoping,YE Zhen. Mean shift research and applications[J]. Control and Decision,2007,22(8):841-847.
[26] 杨能俊,郭宇,方伟光,等. 实时数据驱动的离散制造车间自适应调度方法[J]. 组合机床与自动化加工技术,2020,9(9):175-184. YANG Nengjun,GUO Yu,FANG Weiguang,et al. Real-time data driven adaptive scheduling method of discrete manufacturing workshops[J]. Modular Machine Tool & Automatic Manufacturing Technique,2020,9(9):175-184.

基于改进Q学习的可重入混合流水车间绿色动态调度

An Improved Q Learning Algorithm to Optimize Green Dynamic Scheduling Problem in a Reentrant Hybrid Flow Shop

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	夏连鹏, 权龙, 曹东辉, 尹满义, 刘启明, 李泽鹏. 液气储能双缸驱动大型液压挖掘机动臂节能特性研究[J]. 机械工程学报, 2019, 55(20): 240-248.
[2]	马一太;田华. 制冷空调关键节能减排技术的现状及进展[J]. , 2009, 45(3): 49-56.