基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法

doi:10.3901/JME.2025.03.130

机械工程学报 ›› 2025, Vol. 61 ›› Issue (3): 130-141.doi: 10.3901/JME.2025.03.130

• 特邀专栏：人机联合认知赋能的高端装备设计、制造与运维 • 上一篇

扫码分享

基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法

黄思翰^1,2, 陈建鹏¹, 徐哲¹, 阎艳^1,2, 王国新^1,2

1. 北京理工大学机械与车辆学院北京 100081;
2. 北京理工大学工业知识与数据融合应用工信部重点实验室北京 100081

收稿日期:2024-07-04 修回日期:2024-11-26 发布日期:2025-03-12
作者简介:黄思翰(通信作者)，男，1991年出生，博士，特聘研究员，博士研究生导师。主要研究方向为机器人化智能制造、数字孪生和人工智能。E-mail:hsh@bit.edu.cn;陈建鹏，男，2000年出生。主要研究方向为人机协同作业。E-mail:2393225727@qq.com;徐哲，男，2002年出生。主要研究方向为灵巧手调控。E-mail:lution1019@126.com;阎艳，女，1967年出生，博士，教授，博士研究生导师。主要研究方向为知识工程、智能设计、智能制造。E-mail:yanyan331@bit.edu.cn;王国新，男，1977年出生，博士，教授，博士研究生导师。主要研究方向为知识工程、系统工程、智能制造。E-mail:wangguoxin@bit.edu.cn
基金资助:
北京市自然科学基金重点研究专题(L243009)、国家自然科学基金(52405530)和北京理工大学青年教师学术启动计划资助项目。

Human-robot Autonomous Collaboration Method of Smart Manufacturing Systems Based on Large Language Model and Machine Vision

HUANG Sihan^1,2, CHEN Jianpeng¹, XU Zhe¹, YAN Yan^1,2, WANG Guoxin^1,2

1. School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081;
2. Key Laboratory of Industry Knowledge & Data Fusion Technology and Application, Ministry of Industry and Information Technology, Beijing Institute of Technology, Beijing 100081

Received:2024-07-04 Revised:2024-11-26 Published:2025-03-12

摘要/Abstract

摘要： 工业4.0阶段，人工智能、大数据、物联网等新兴技术层出不穷，加速推动制造业转型升级，在这个过程中，各类机器人扮演着越来越重要的角色，也为智能制造高质量发展夯实了基础。随着工业5.0的提出，以人为中心的理念逐渐深入人心，催生了人本智造这一新兴领域。人与机器人在智能制造系统中的界限变得越来越不明显，人与机器人自主协同作业研究成为了热点。因此，提出基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法，借助机器视觉和大语言模型的优势提高智能制造系统中人机协同作业的智能化水平。首先，融合机器视觉和深度学习对智能制造系统中人机协同作业过程进行动态精准感知，通过融合YOLO算法和迁移学习来识别作业状态，利用长短期记忆网络和注意力机制准确追踪操作工动作。然后，面向人机协同作业对大语言模型进行微调，建立基于微调大模型的人机协同作业决策框架，为提供机器人自主配合操作工完成动态作业的任务指令，形成人机自主协同作业闭环。最后，通过一个减速器装配案例验证了该方法的有效性。

关键词: 工业5.0, 人本智造, 智能制造系统, 大语言模型, 机器视觉, 人机自主协同作业, 深度学习

Abstract: In Industry 4.0, the emerging technologies such as artificial intelligence, big data, and the Internet of Things are appearing endlessly, accelerating the transformation and upgrading of the manufacturing industry. In this process, industry robot plays an increasingly important role, which also lays a solid foundation for the high-quality development of intelligent/smart manufacturing. With the proposal of Industry 5.0, human centricity concept becomes popular, which has given birth to the emerging field of human-centric smart manufacturing. The boundary between human and robot in the smart manufacturing systems gets blurred, and the research on human-robot autonomous collaboration has attracted more and more attentions. Therefore, proposes a human-robot autonomous collaboration method based on large language model and machine vision to improve the intelligence level of human-robot collaboration. First, dynamic perception of the working process for human-robot collaboration is carried out by the fusion of machine vision and deep learning, where the fusion of YOLO and transfer learning is adopted to accurately identify the operate progress and the long short-term memory network and attention mechanism are combined to recognize the actions of operator. Second, the large language model is fine-tuned for human-robot collaboration to realize autonomous operating decision for smart robot during the dynamic work process. Finally, a reducer assembly case is used to verify the effectiveness of the proposed method.

Key words: Industry 5.0, human-centric smart manufacturing, smart manufacturing systems, large language model, machine vision, human-robot autonomous collaboration, deep learning

中图分类号:

黄思翰, 陈建鹏, 徐哲, 阎艳, 王国新. 基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法[J]. 机械工程学报, 2025, 61(3): 130-141.

HUANG Sihan, CHEN Jianpeng, XU Zhe, YAN Yan, WANG Guoxin. Human-robot Autonomous Collaboration Method of Smart Manufacturing Systems Based on Large Language Model and Machine Vision[J]. Journal of Mechanical Engineering, 2025, 61(3): 130-141.

参考文献

[1] LASI H，FETTKE P，KEMPER H G，et al. Industry 4.0[J]. Business & Information Systems Engineering，2014，6:239-242.
[2] HUANG S，WANG B，LI X，et al. Industry 5.0 and society 5.0—Comparison，complementation and co-evolution[J]. Journal of Manufacturing Systems，2022，64:424-428.
[3] WANG B，ZHOU H，YANG G，et al. Human digital twin (HDT) driven human-cyber-physical systems:Key technologies and applications[J]. Chinese Journal of Mechanical Engineering，2022，35(1):11.
[4] HE Q，LI L，LI D，et al. From digital human modeling to human digital twin: Framework and perspectives in human factors[J]. Chinese Journal of Mechanical Engineering，2024，37(1):9.
[5] 马南峰，姚锡凡，陈飞翔，等. 面向工业5.0的人本智造[J]. 机械工程学报，2022，58(18):88-102. MA Nanfeng，YAO Xifan，CHEN Feixiang，et al. Human-centric smart manufacturing for industry 5.0[J]. Journal of Mechanical Engineering，2022，58(18):88-102.
[6] AJOUDANI A，ZANCHETTIN A M，IVALDI S，et al. Progress and prospects of the human–robot collaboration[J]. Autonomous Robots，2018，42:957-975.
[7] LIU H，WANG L. Human motion prediction for human-robot collaboration[J]. Journal of Manufacturing Systems，2017，44:287-294.
[8] SHI Y，SHEN W，WANG L，et al. A cognitive digital twins framework for human-robot collaboration[J]. Procedia Computer Science，2022，200:1867-1874.
[9] 黄思翰，王柏村，张美迪，等. 面向人本智造的新一代操作工:参考架构，使能技术与典型场景[J]. 机械工程学报，2022，58(18):251-264. HUANG Sihan，WANG Baicun，ZHANG Meidi，et al. Operator 4.0 towards human-centric smart manufacturing:Framework，enabling technologies and typical scenarios [J]. Journal of Mechanical Engineering，2022，58(18):251-264.
[10] HALME R J，LANZ M，KÄMÄRÄINEN J，et al. Review of vision-based safety systems for human-robot collaboration[J]. Procedia CIRP，2018，72:111-116.
[11] WEISS A，WORTMEIER A K，KUBICEK B. Cobots in industry 4.0:A roadmap for future practice studies on human-robot collaboration[J]. IEEE Transactions on Human-machine Systems，2021，51(4):335-345.
[12] WANG P，LIU H，WANG L，et al. Deep learning-based human motion recognition for predictive context-aware human-robot collaboration[J]. CIRP Annals，2018，67(1):17-20.
[13] AL-AMIN M，QIN R，MONIRUZZAMAN M，et al. An individualized system of skeletal data-based CNN classifiers for action recognition in manufacturing assembly[J]. Journal of Intelligent Manufacturing，2023:1-17.
[14] BILBERG A，MALIK A A. Digital twin driven human–robot collaborative assembly[J]. CIRP Annals，2019，68(1):499-502.
[15] ZHU Q，HUANG S，WANG G，et al. Dynamic reconfiguration optimization of intelligent manufacturing system with human-robot collaboration based on digital twin[J]. Journal of Manufacturing Systems，2022，65:330-338.
[16] LIU H，WANG L. Gesture recognition for human-robot collaboration:A review[J]. International Journal of Industrial Ergonomics，2018，68:355-367.
[17] DUAN J，FANG Y，ZHANG Q，et al. HRC for dual-robot intelligent assembly system based on multimodal perception[J]. Proceedings of the Institution of Mechanical Engineers，Part B:Journal of Engineering Manufacture，2024，238(4):562-576.
[18] LI Y，GE S S. Human–robot collaboration based on motion intention estimation[J]. IEEE/ASME Transactions on Mechatronics，2013，19(3):1007-1014.
[19] 李祥文，宋程，丁帅. 人机协同决策中的人因能力评估研究[J]. 中国管理科学，2024，32(3):145-155. LI Xiangwen，SONG Cheng，DING Shuai. Human factors capability assessment in human-machine collaborative decision-making[J]. Chinese Journal of Management Science，2024，32(3):145-155.
[20] YE Y，YOU H，DU J. Improved trust in human-robot collaboration with ChatGPT[J]. IEEE Access，2023，11:55748-55754.
[21] WANG L，MA C，FENG X，et al. A survey on large language model based autonomous agents[J]. Frontiers of Computer Science，2024，18(6):186345.
[22] WANG C Y，BOCHKOVSKIY A，LIAO H Y M. YOLOv7:Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023:7464-7475.
[23] 庄福振，罗平，何清，等. 迁移学习研究进展[J]. 软件学报，2014，26(1):26-39. ZHUANG Fuzhen，LUO Ping，HE Qing，et al. Survey on transfer learning research[J]. Journal of Software，2014，26(1):26-39.
[24] WANG Y，HUANG M，ZHU X，et al. Attention-based LSTM for aspect-level sentiment classification[C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016:606-615.
[25] KIM C Y，LEE C P，MUTLU B. Understanding large-language model (LLM)-powered human-robot interaction[C]//Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction. 2024:371-380.
[26] 王湉，范峻铭，郑湃. 基于大语言模型的人机交互移动检测机器人导航方法[J]. 计算机集成制造系统，2024，30(5):1587-1594. WANG Tian，FAN Junming，ZHENG Pai. Large language model-based approach for human-mobile inspection robot interactive navigation[J]. Computer Integrated Manufacturing Systems，2024，30(5):1587-1594.

基于大语言模型和机器视觉的智能制造系统人机自主协同作业方法

Human-robot Autonomous Collaboration Method of Smart Manufacturing Systems Based on Large Language Model and Machine Vision

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李嘉佳, 易茜, 冯毅雄, 朱鹏兴, 易树平. 人本智造单元中人-智能系统协同双智能体工作机制研究[J]. 机械工程学报, 2025, 61(3): 105-118.
[2]	李春凯, 王嘉昕, 石玗, 代悦. GTAW熔池激光条纹动态行为与熔透预测方法研究[J]. 机械工程学报, 2024, 60(6): 236-244.
[3]	蒋周明矩, 熊异, 王柏村. 面向工业5.0的人机协作增材制造[J]. 机械工程学报, 2024, 60(3): 238-253.
[4]	史丽晨, 史炜椿, 王海涛, 李金阳, 刘亚雄. 基于DRSN-BiLSTM模型的刀具磨损预测方法研究[J]. 机械工程学报, 2024, 60(24): 66-74.
[5]	赵德望, 姜超, 赵延广, 杨文平, 樊俊铃. 铝锂合金疲劳性能及预测研究-基于试验和“浅层”+“深度”混合神经网络方法[J]. 机械工程学报, 2024, 60(16): 190-199.
[6]	罗晨, 邵佳丰, 王伟, 张刚, 周怡君. 考虑镜面反射和阴影的复杂曲面测量视点规划[J]. 机械工程学报, 2024, 60(15): 304-315.
[7]	付玲, 佘玲娟, 颜镀镭, 张鹏, 龙湘云. 基于内嵌物理信息与注意力机制BiLSTM神经网络的臂架系统疲劳损伤预测模型[J]. 机械工程学报, 2024, 60(13): 205-215.
[8]	陈钱, 陈康康, 董兴建, 皇甫一樊, 彭志科, 孟光. 一种面向机械设备故障诊断的可解释卷积神经网络[J]. 机械工程学报, 2024, 60(12): 65-76.
[9]	刘岳开, 王天杨, 褚福磊. 基于低延迟可解释性深度学习的复杂旋转机械关键部件知识嵌入与诊断方法研究[J]. 机械工程学报, 2024, 60(12): 107-115.
[10]	赖旭伟, 丁昆, 张楷, 黄锋飞, 郑庆, 李致萱, 丁国富. 基于可解释物理引导空间注意力改进的跨工艺参数立铣刀磨损辨识[J]. 机械工程学报, 2024, 60(12): 147-157.
[11]	林昕, 毛杨坤, 傅盈西, 朱锟鹏. 基于图像语义分割与图卷积的选区激光熔融成形过程羽流运动特征分析[J]. 机械工程学报, 2024, 60(12): 168-182.
[12]	邵海东, 肖一鸣, 邓乾旺, 任颖莹, 韩特. 基于不确定性感知网络的可信机械故障诊断[J]. 机械工程学报, 2024, 60(12): 194-206.
[13]	李玖法, 邹博文, 任玥. 考虑车-路交互作用的车辆轨迹预测算法研究[J]. 机械工程学报, 2024, 60(10): 76-85.
[14]	许思远, 张卫东, 毛玉明, 牛斌. 数据驱动的指定变形非均质多胞结构优化设计[J]. 机械工程学报, 2024, 60(1): 85-95.
[15]	何赟泽, 李响, 王洪金, 侯岳骏, 张帆, 牟欣颖, 刘浩, 程豪, 李时华, 李杰. 基于可见光和热成像的风机叶片全周期无损检测综述[J]. 机械工程学报, 2023, 59(6): 32-45.