Study on the Discrete Manufacturing Workshop Scheduling Method Based on DQN Algorithm Considering AGV

doi:10.3901/JME.2024.18.338

Abstract

Abstract: For the production scheduling of discrete manufacturing workshops, it is not only necessary to determine the processing machine of each process of the job and the processing sequence of the processes on the machine, but also according to the job scheduling plan, the AGV needs to transport each job to the corresponding machine for processing before the specified time point, In order to improve the execution rate of the scheduling scheme, a discrete manufacturing workshop scheduling model is constructed that considers constraints such as workshop machine layout, job process route, AGV handling time and AGV position, and minimizes job completion time and AGV load balance as comprehensive goals. Build a reinforcement learning environment based on the discrete manufacturing workshop scheduling mathematical model, including the state space of job, machine and car, scheduling decision action space and reward function; based on the established reinforcement learning environment, design a job car scheduling method based on DQN algorithm, and design job agent, read the local environment of the workshop, map the local environment to the weight of the relevant parameters of the job, and obtain the job scheduling list according to the weight to realize the action selection from the workshop state to the job scheduling. Design the AGV agent, and obtain the relevant parameters of the AGV handling by reading the scheduling decision and workshop information of the job agent, and realize the interaction between the AGV agent and the job agent. The handling related parameters and the car status information in the local environment of the workshop are mapped into the relevant weights of the car scheduling, and the car scheduling list is obtained according to the weights to realize the action selection of the car scheduling. Finally, the algorithm is tested through the actual case of discrete manufacturing workshop. The test results show that the scheduling algorithm based on the DQN algorithm can effectively solve the discrete manufacturing workshop scheduling problem considering the handling of AGVs, minimize the maximum completion time of job, and balance the handling of AGVs load.

Key words: discrete manufacturing shop, job scheduling, AGV scheduling, DQN algorithm

CLC Number:

TP301

ZHOU Yaqin, XIAO Meng, Lü Zhijun, WANG Junliang, ZHANG Jie. Study on the Discrete Manufacturing Workshop Scheduling Method Based on DQN Algorithm Considering AGV[J]. Journal of Mechanical Engineering, 2024, 60(18): 338-348.

References

[1] 蒋静静. 基于深度强化学习的离散型制造企业车间动态调度研究[D]. 西安：西安理工大学，2020. JIANG Jingjing. Research on Dynamic job shop scheduling of discrete manufacturing enterprises based on deep reinforcement learning[D]. Xi’an：Xi’an University of Technology，2020.
[2] 甄文冬，陈进. 制造车间生产能力的影响因素研究[J]. 轻工机械，2020，38(2)：99-102，107. ZHEN Wendong，CHEN Jin. Study on Influencing Factors of production capacity in manufacturing workshop[J]. Light Industry Machinery，2020，38(2)：99-102，107.
[3] 黎书文，张成龙，周知进. 基于改进粒子群算法的离散制造车间柔性调度优化[J]. 组合机床与自动化加工技术，2018(11)：150-152. LI Shuwen，ZHANG Chenglong，ZHOU Zhijin. Flexible scheduling optimization of discrete manufacturing workshop based on improved particle swarm optimization algorithm[J]. Modular Machine Tool & Automatic Manufacturing Technique，2018(11)：150-152.
[4] PIROZMAND P，HOSSEINABADI A，FARROKHZAD M，et al. Multi-objective hybrid genetic algorithm for task scheduling problem in cloud computing[J]. Neural Computing & Applications，2021(33)：13075-13088.
[5] SZ A，XIANG L A，BZ A，et al. Multi-objective optimisation in flexible assembly job shop scheduling using a distributed ant colony system[J]. European Journal of Operational Research，2020，283(2)：441-460.
[6] MAO C L. Production management of multi-objective flexible job-shop based on improved PSO[J]. International Journal of Simulation Modelling，2021，20(2)：422-433.
[7] 曾创锋，刘建军，陈庆新，等. 求解一类无关并行机调度的遗传迭代贪心算法[J]. 工业工程，2021，24(2)：110-118. ZENG Chuangfeng，LIU Jianjun，CHEN Qingxin，et al. Genetic iterative greedy algorithm for scheduling a class of unrelated parallel machines[J]. Industrial Engineering Journal，2021，24(2)：110-118.
[8] 李雯璐，赵秀栩. 求解不相关并行机调度问题的十进制多目标灰狼算法[J]. 计算机应用研究，2021，38(10)：3067-3071. LI Wenlu，ZHAO Xiuxu，Decimal multi-objective grey wolf algorithm for unrelated parallel machine scheduling problem[J]. Application Research of Computers，2021，38(10)：3067-3071.
[9] 谢志强，夏迎春. 基于遗传算法和分枝定界的多车间空闲产能调度方法[J]. 机械工程学报，2022，58(22)：462-472. XIE Zhiqiang，XIA Yingchun. Multi-shop idle capacity scheduling method based on genetic algorithmand branch and bound[J]. Journal of Mechnical Engineering， 2022，58(22)：462-472.
[10] 郑重. 自动化集装箱码头AGV调度及路径优化[J]. 中国港口，2021(3)：34-36. ZHENG Zhong. AGV scheduling and route optimization in automated container terminal[J]. China Ports，2021(3)：34-36.
[11] 郭沛佩，付建林，江海凡，等. 基于规则的柔性作业车间机床与AGV联合调度优化[J]. 制造技术与机床，2021(9)：107-113. GUO Peipei，FU Jianlin，JIANG Haifan，et al. Rule based joint scheduling optimization of machine tool and AGV in flexible job shop[J]. Manufacturing Technology & Machine Tool，2021(9)：107-113.
[12] ABBEEL P，COATES A，QUIGLEY M，et al. An application of reinforcement learning to aerobatic helicopter flight[C]//Advances in Neural Information Processing Systems，2007：1-8.
[13] VOLODYMYR M，KORAY K，DAVID S，et al. Playing atari with deep reinforcement learning[EB/OL]. https://arxiv.org/pdf/1312.5602.pdf.
[14] CUNHA B，MADUREIRA A M，FONSECA B，et al. Deep reinforcement learning as a job shop scheduling solver：A literature review[C]//International Conference on Hybrid Intelligent Systems，2018：350-359.
[15] SUTTON R S，BARTO A G. Reinforcement learning：An introduction[J]. Cambridge：MIT Press，1998.
[16] LIU C L，CHANG C C，TSENG C J. Actor-critic deep reinforcement learning for solving job shop scheduling problems[J]. IEEE Access，2020(8)：71752-71762.
[17] WANG J，HE J，ZHANG J. A reinforcement learning method to optimize the priority of product for scheduling the large-scale complex manufacturing systems[C]//The 48th International Conference on Computers & Industrial Engineering，2018：2-5.
[18] 贺俊杰，张洁，张朋，等. 基于长短期记忆近端策略优化强化学习的等效并行机在线调度方法[J]. 中国机械工程，2022，33(3)：329-338. HE Junjie，ZHANG Jie，ZHANG Peng，et al. On line scheduling method of equivalent parallel machine based on short-term memory near end strategy optimization and reinforcement learning[J]. China Mechanical Engineering，2022，33(3)：329-338.
[19] LUO S. Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning[J]. Applied Soft Computing，2020，91：106208.
[20] YANG S，XU Z，WANG J. Intelligent decision-making of scheduling for dynamic permutation flowshop via deep reinforcement learning[J]. Sensors，2021，21(3)：1019.
[21] 马铭阳. 柔性作业车间加工机器与配送AGV双资源集成调度问题[D]. 长春：吉林大学，2021. MA Mingyang. The dual resource integration scheduling problem of processing machine and distribution AGV in flexible job shop[D]. Changchun：Jilin University，2021.
[22] BILGE U，ULUSOY G. A time window approach to simultaneous scheduling of machines and material handling system in an FMS[J]. Operations Research，1995，43(6)：1058-1070.
[23] ABDELMAGUID T F，NASSEF A O，KAMAL B A，et al. A hybrid GA/heuristic approach to the simultaneous scheduling of machines and automated guided vehicles[J]. International Journal of Production Research，2004，43(2)：267-281.
[24] KUMAR M V S，JANARDHANA R，RAO C S P. Simultaneous scheduling of machines and vehicles in an FMS environment with alternative routing[J]. The International Journal of Advanced Manufacturing Technology，2011，53(1)：339-351.
[25] SAHIN C，DEMIRTAS M，EROL R，et al. A multi-agent based approach to dynamic scheduling with flexible processing capabilities[J]. Journal of Intelligent Manufacturing，2017，28(8)：1827-1845.
[26] 周亚勤，杨长祺，吕佑龙，等. 双资源约束的航天结构件车间生产调度方法[J]. 机械工程学报，2018，54(9)：55-63. ZHOU Yaqin，YANG Changqi，LÜ Youlong，et al，Production scheduling method for aerospace structural parts workshop with dual resource constraints[J]. Journal of Mechanical Engineering，2018，54(9)：55-63.